{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XGBoost Parameter Tuning for Happy Customer Bank"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7.调整树的参数：降低学习率，调整树的树木"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据说明:\n",
    "数据集共26个字段: 其中1-24列为输入特征，25-26列为输出特征。\n",
    "    1. ID - 唯一ID（不能用于预测）\n",
    "    2. Gender - 性别\n",
    "    3. City - 城市\n",
    "    4. Monthly_Income - 月收入（以卢比为单位）\n",
    "    5. DOB - 出生日期\n",
    "    6. Lead_Creation_Date - 潜在（贷款）创建日期\n",
    "    7. Loan_Amount_Applied - 贷款申请请求金额（印度卢比，INR）\n",
    "    8. Loan_Tenure_Applied - 贷款申请期限（单位为年）\n",
    "    9. Existing_EMI -现有贷款的EMI（EMI：电子货币机构许可证） \n",
    "    10. Employer_Name雇主名称\n",
    "    11. Salary_Account - 薪资帐户银行\n",
    "    12. Mobile_Verified - 是否移动验证（Y / N）\n",
    "    13. VAR5 - 连续型变量\n",
    "    14. VAR1-  类别型变量\n",
    "    15. Loan_Amount_Submitted - 提交的贷款金额（在看到资格后修改和选择）\n",
    "    16. Loan_Tenure_Submitted - 提交的贷款期限（单位为年，在看到资格后修改和选择）\n",
    "    17. Interest_Rate - 提交贷款金额的利率\n",
    "    18. Processing_Fee - 提交贷款的处理费（INR）\n",
    "    19. EMI_Loan_Submitted -提交的EMI贷款金额（INR）\n",
    "    20. Filled_Form - 后期报价后是否已填写申请表格\n",
    "    21. Device_Type - 进行申请的设备（浏览器/移动设备）\n",
    "    22. Var2 - 类别型变量\n",
    "    23. Source - 类别型变量\n",
    "    24. Var4 - 类别型变量\n",
    "\n",
    "输出：\n",
    "    25. LoggedIn - 是否login（只用于理解问题的变量，不能用于预测，测试集中没有）\n",
    "    26. Disbursed - 是否发放贷款（目标变量），1为发放贷款（目标客户）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = pd.read_csv('train_save.csv')\n",
    "test = pd.read_csv('test_save.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((87020, 52), (37717, 51))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.shape, test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEKCAYAAADaa8itAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAFpFJREFUeJzt3Xu0JWV95vHvw02Qi4C2LtNAGmNHJTreWoImEy8oAk5oZEGCyxEGmcFkAHESHTHJsuNdEoRI4iWsQLiMw0UGtEWUQQSdOIo0F8UGWXQwQg8EWkFEGGEafvPHfg/s7j6Xorv2OWzP97PWXqfqrbdq/7ar5VlVb9VbqSokSerDZnNdgCTpV4ehIknqjaEiSeqNoSJJ6o2hIknqjaEiSeqNoSJJ6o2hIknqjaEiSerNFnNdwGx7xjOeUYsWLZrrMiRpbFxzzTU/qaoFXfrOu1BZtGgRK1asmOsyJGlsJPlx175e/pIk9cZQkST1xlCRJPXGUJEk9cZQkST1xlCRJPXGUJEk9cZQkST1xlCRJPVm3j1Rv6le/p6z5roEPQld89eHzXUJ0pOCZyqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTejDRUkvyXJCuT/CDJOUm2TrJ7kquS3JLkvCRbtb5Paeur2vZFQ8d5X2u/Ockbh9r3bW2rkhw/yt8iSZrZyEIlyULgncCSqnohsDlwKHACcHJVLQbuBY5suxwJ3FtVzwVObv1Iskfb77eAfYFPJ9k8yebAp4D9gD2At7S+kqQ5MurLX1sA2yTZAngqcCfwOuCCtv1M4MC2vLSt07bvnSSt/dyqeqiqfgSsAvZsn1VVdWtVPQyc2/pKkubIyEKlqv4PcCJwG4MwuQ+4BvhZVa1t3VYDC9vyQuD2tu/a1v/pw+3r7TNVuyRpjozy8tdODM4cdgd+DdiWwaWq9dXELlNse6Ltk9VyVJIVSVasWbNmptIlSRtplJe/Xg/8qKrWVNX/Ay4EXgXs2C6HAewC3NGWVwO7ArTtTwPuGW5fb5+p2jdQVadW1ZKqWrJgwYI+fpskaRKjDJXbgL2SPLWNjewN3AhcARzc+hwOfLEtL2/rtO1fr6pq7Ye2u8N2BxYD3wWuBha3u8m2YjCYv3yEv0eSNIORvaSrqq5KcgFwLbAWuA44FfgycG6SD7e209oupwFnJ1nF4Azl0HaclUnOZxBIa4Gjq+oRgCTHAJcyuLPs9KpaOarfI0ma2Ujf/FhVy4Bl6zXfyuDOrfX7/hI4ZIrjfAT4yCTtlwCXbHqlkqQ++ES9JKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3M4ZKkm2TbNaWfzPJAUm2HH1pkqRx0+VM5ZvA1kkWApcDRwBnjLIoSdJ46hIqqaoHgYOAv62qNwN7jLYsSdI46hQqSV4JvBX4cmvbYnQlSZLGVZdQeRfwPuCiqlqZ5DnAFaMtS5I0jmY846iqbwDfSLJtW78VeOeoC5MkjZ8ud3+9MsmNwE1t/cVJPj3yyiRJY6fL5a+/Ad4I/BSgqr4H/N4oi5IkjadODz9W1e3rNT0yglokSWOuy11ctyd5FVBJtmIwnnLTaMuSJI2jLmcqfwQcDSwEVgMvaeuSJK2jy91fP2HwjIokSdPqcvfXmUl2HFrfKcnpoy1LkjSOulz++jdV9bOJlaq6F3jp6EqSJI2rLqGyWZKdJlaS7IzTtEiSJtElHD4B/O8kF7T1Q4CPjK4kSdK46jJQf1aSa4DXAgEOqqobR16ZJGnsdH3z4w+BC4EvAr9IsluXnZLsmOSCJD9MclOb8mXnJJcluaX93an1TZJTkqxK8v0kLxs6zuGt/y1JDh9qf3mSG9o+pyRJ958uSepbl7u/jgXuAi4DLmYw/f3FHY//SeCrVfV84MUMHpo8Hri8qhYzeOnX8a3vfsDi9jkK+Ez7/p2BZcBvA3sCy4bGeD7T+k7st2/HuiRJI9BlTOU44HlV9dMncuAkOzCYI+w/AFTVw8DDSZYCr2ndzgSuBN4LLAXOqqoCvtPOcp7d+l5WVfe0414G7JvkSmCHqvp2az8LOBD4yhOpU5LUny6Xv24H7tuIYz8HWAP8Y5LrkvxDmz7/WVV1J0D7+8zWf2H7rgmrW9t07asnaZckzZEuZyq3Alcm+TLw0ERjVZ3U4dgvA46tqquSfJLHL3VNZrLxkNqI9g0PnBzF4DIZu+3WaThIkrQRupyp3MZgPGUrYPuhz0xWA6ur6qq2fgGDkLmrXdai/b17qP+uQ/vvAtwxQ/suk7RvoKpOraolVbVkwYIFHUqXJG2MLrcUfwAgybZV9UDXA1fVvya5PcnzqupmYG/gxvY5HPh4+/vFtsty4Jgk5zIYlL+vqu5Mcinw0aHB+X2A91XVPUnuT7IXcBVwGPC3XeuTJPVvxlBJ8krgNGA7YLckLwbeUVX/ucPxjwU+16bMvxU4gsHZ0flJjmRwFnRI63sJsD+wCniw9aWFx4eAq1u/D04M2gN/DJwBbMNggN5BekmaQ13GVCbe/LgcBm9+TNLpzY9VdT2wZJJNe0/St5hiSv2qOh3YYBLLqloBvLBLLZKk0fPNj5Kk3vjmR0lSb3zzoySpN9OeqSTZHHhbVfnmR0nSjKY9U6mqRxhMnyJJ0oy6jKl8K8nfAecBjz2nUlXXjqwqSdJY6hIqr2p/PzjUVsDr+i9HkjTOZhpT2Qz4TFWdP0v1SJLG2ExjKo8Cx8xSLZKkMdflluLLkrw7ya7trY07txdnSZK0ji5jKm9vf4efTSkG70uRJOkxXWYp3n02CpEkjb8usxQfNll7VZ3VfzmSpHHW5fLXK4aWt2Yww/C1gKEiSVpHl8tfxw6vJ3kacPbIKpIkja1OU9+v50Fgcd+FSJLGX5cxlS8xuNsLBiG0B+DDkJKkDXQZUzlxaHkt8OOqWj2ieiRJY6xLqNwG3FlVvwRIsk2SRVX1LyOtTJI0drqMqXweeHRo/ZHWJknSOrqEyhZV9fDESlveanQlSZLGVZdQWZPkgImVJEuBn4yuJEnSuOoypvJHwOfai7pg8J76SZ+ylyTNb10efvxnYK8k2wGpqvtHX5YkaRzNePkryUeT7FhVv6iq+5PslOTDs1GcJGm8dBlT2a+qfjaxUlX3AvuPriRJ0rjqEiqbJ3nKxEqSbYCnTNNfkjRPdRmo/2/A5Un+kcF0LW8HzhxpVZKksdRloP6vknwfeH1r+lBVXTrasiRJ46jLmQrAdcCWDM5UrhtdOZKkcdbl7q8/AL4LHAz8AXBVkoNHXZgkafx0OVP5c+AVVXU3QJIFwNeAC0ZZmCRp/HS5+2uziUBpftpxP0nSPNPlTOWrSS4FzmnrfwhcMrqSJEnjqsvdX+9JchDwu0CAU6vqopFXJkkaO53u/qqqC4ELR1yLJGnMOTYiSeqNoSJJ6s2UoZLk8vb3hNkrR5I0zqYbU3l2klcDByQ5l8Eg/WOq6tqRViZJGjvTXf56P3A8sAtwEvCJoc+JXb8gyeZJrktycVvfPclVSW5Jcl6SrVr7U9r6qrZ90dAx3tfab07yxqH2fVvbqiTHd//ZkqRRmDJUquqCqtoP+Kuqeu16n9c9ge84DrhpaP0E4OSqWgzcCxzZ2o8E7q2q5wInt34k2QM4FPgtYF/g0y2oNgc+BewH7AG8pfWVJM2RGQfqq+pDSQ5IcmL7/LuuB0+yC/Am4B/aeoDX8fgUL2cCB7blpTw+pf4FwN6t/1Lg3Kp6qKp+BKwC9myfVVV1a1U9DJzb+kqS5kiXCSU/xuBs48b2Oa61dfE3wH8FHm3rTwd+VlVr2/pqYGFbXgjcDtC239f6P9a+3j5TtU/2G45KsiLJijVr1nQsXZL0RHW5pfhNwBuq6vSqOp3BJag3zbRTO6O5u6quGW6epGvNsO2Jtm/YWHVqVS2pqiULFiyYpmpJ0qbo+j6VHYF72vLTOu7zOwzuHNsf2BrYgcGZy45JtmhnI7sAd7T+q4FdgdVJtmjfc89Q+4ThfaZqlyTNgS5nKh8DrktyRpIzgWuAj860U1W9r6p2qapFDAbav15VbwWuYPBuFoDDgS+25eVtnbb961VVrf3QdnfY7sBiBu93uRpY3O4m26p9x/IOv0eSNCJdJpQ8J8mVwCsYXHJ6b1X96yZ853uBc5N8mMFbJE9r7acBZydZxeAM5dD2/SuTnM9gPGctcHRVPQKQ5BjgUmBz4PSqWrkJdUmSNlHXCSXvZBPOAqrqSuDKtnwrgzu31u/zS+CQKfb/CPCRSdovwWn4JelJw7m/JEm9MVQkSb2ZNlSSbJbkB7NVjCRpvE0bKlX1KPC9JLvNUj2SpDHWZaD+2cDKJN8FHphorKoDRlaVJGksdQmVD4y8CknSr4Quz6l8I8mvA4ur6mtJnsrguRBJktbRZULJ/8Rg1uC/b00LgS+MsihJ0njqckvx0Qzm8fo5QFXdAjxzlEVJksZTl1B5qL2vBIA22eOkswFLkua3LqHyjSR/BmyT5A3A54EvjbYsSdI46hIqxwNrgBuAdzCYa+svRlmUJGk8dbn769E25f1VDC573dympJckaR0zhkqSNwGfBf6ZwdT3uyd5R1V9ZdTFSZLGS5eHHz8BvLaqVgEk+Q3gy4ChIklaR5cxlbsnAqW5Fbh7RPVIksbYlGcqSQ5qiyuTXAKcz2BM5RAGr/KVJGkd013++v2h5buAV7flNcBOI6tIkjS2pgyVqjpiNguRJI2/Lnd/7Q4cCywa7u/U95Kk9XW5++sLwGkMnqJ/dLTlSJLGWZdQ+WVVnTLySiRJY69LqHwyyTLgfwIPTTRW1bUjq0qSNJa6hMqLgLcBr+Pxy1/V1iVJekyXUHkz8Jzh6e8lSZpMlyfqvwfsOOpCJEnjr8uZyrOAHya5mnXHVLylWJK0ji6hsmzkVUiSfiV0eZ/KN2ajEEnS+OvyRP39PP5O+q2ALYEHqmqHURYmSRo/Xc5Uth9eT3IgsOfIKpIkja0ud3+to6q+gM+oSJIm0eXy10FDq5sBS3j8cpgkSY/pcvfX8HtV1gL/AiwdSTWSpLHWZUzF96pIkjqZ7nXC759mv6qqD42gHknSGJvuTOWBSdq2BY4Eng4YKpKkdUz3OuFPTCwn2R44DjgCOBf4xFT7SZLmr2nHVJLsDPwJ8FbgTOBlVXXvbBQmSRo/Uz6nkuSvgauB+4EXVdVfPpFASbJrkiuS3JRkZZLjWvvOSS5Lckv7u1NrT5JTkqxK8v0kLxs61uGt/y1JDh9qf3mSG9o+pyTJRvxvIEnqyXQPP/4p8GvAXwB3JPl5+9yf5Ocdjr0W+NOqegGwF3B0kj2A44HLq2oxcHlbB9gPWNw+RwGfgcfOlpYBv83gSf5lE0HU+hw1tN++3X62JGkUpgyVqtqsqrapqu2raoehz/Zd5v2qqjsnXjlcVfcDNwELGTzjcmbrdiZwYFteCpxVA98BdkzybOCNwGVVdU87U7oM2Ldt26Gqvl1VBZw1dCxJ0hx4wtO0bIwki4CXAlcBz6qqO2EQPMAzW7eFwO1Du61ubdO1r56kfbLvPyrJiiQr1qxZs6k/R5I0hZGHSpLtgP8BvKuqprtsNtl4SG1E+4aNVadW1ZKqWrJgwYKZSpYkbaSRhkqSLRkEyueq6sLWfFe7dEX7e3drXw3sOrT7LsAdM7TvMkm7JGmOjCxU2p1YpwE3VdVJQ5uWAxN3cB0OfHGo/bB2F9hewH3t8tilwD5JdmoD9PsAl7Zt9yfZq33XYUPHkiTNgS4TSm6s3wHeBtyQ5PrW9mfAx4HzkxwJ3AYc0rZdAuwPrAIeZPCgJVV1T5IPMbi9GeCDVXVPW/5j4AxgG+Ar7SNJmiMjC5Wq+icmH/cA2HuS/gUcPcWxTgdOn6R9BfDCTShTktSjWbn7S5I0PxgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN6Mfagk2TfJzUlWJTl+ruuRpPlsi7kuYFMk2Rz4FPAGYDVwdZLlVXXj3FYmzY3bPviiuS5BT0K7vf+GWfuucT9T2RNYVVW3VtXDwLnA0jmuSZLmrXEPlYXA7UPrq1ubJGkOjPXlLyCTtNUGnZKjgKPa6i+S3DzSquaPZwA/mesingxy4uFzXYI25L/PCcsm+0/lE/LrXTuOe6isBnYdWt8FuGP9TlV1KnDqbBU1XyRZUVVL5roOaTL++5wb437562pgcZLdk2wFHAosn+OaJGneGuszlapam+QY4FJgc+D0qlo5x2VJ0rw11qECUFWXAJfMdR3zlJcU9WTmv885kKoNxrUlSdoo4z6mIkl6EjFUNKOZpsJJ8pQk57XtVyVZNPtVaj5KcnqSu5P8YIrtSXJK+7f5/SQvm+0a5xtDRdMamgpnP2AP4C1J9liv25HAvVX1XOBk4ITZrVLz2BnAvtNs3w9Y3D5HAZ+ZhZrmNUNFM+kyFc5S4My2fAGwd5JNftpKmklVfRO4Z5ouS4GzauA7wI5Jnj071c1Phopm0mUqnMf6VNVa4D7g6bNSnTQ9p3KaZYaKZtJlKpxO0+VIc8B/m7PMUNFMukyF81ifJFsAT2P6SxLSbOk0lZP6Y6hoJl2mwlkOTMyoeDDw9fIBKD05LAcOa3eB7QXcV1V3znVRv8rG/ol6jdZUU+Ek+SCwoqqWA6cBZydZxeAM5dC5q1jzSZJzgNcAz0iyGlgGbAlQVZ9lMNvG/sAq4EHgiLmpdP7wiXpJUm+8/CVJ6o2hIknqjaEiSeqNoSJJ6o2hIknqjaEidZTkkSTXJ1mZ5HtJ/iTJZm3bkiSnTLPva5JcPHvVbvD9f5nk3XP1/Zo/fE5F6u7/VtVLAJI8E/jvDGYPWFZVK4AVo/riJFu0edWkJzXPVKSNUFV3M5hK/Zj2tPZjZyJJXt3OaK5Pcl2S7dtuOyS5KMmNST47dJbzi4njJjk4yRlt+YwkJyW5AjhhquMmeU+Sq9v7Qj4wdKw/b+/B+RrwvNn430XyTEXaSFV1awuGZ6636d3A0VX1rSTbAb9s7XsyeCfNj4GvAgcxeFXAdH4TeH1VPZLkS+sfN8k+DN4VsieDyROXJ/k94AEGMxu8lMH/z68Frtm0XyzNzDMVadNMNgvut4CTkrwT2HHostV323tpHgHOAX63w/E/3/pPddx92uc6BsHxfAYh82+Bi6rqwar6ORvO1yaNhKEibaQkzwEeAe4ebq+qjwP/EdgG+E6S509sWu8QNUn71uv1eWCG4wb4WFW9pH2eW1WnTfF90sgZKtJGSLIA+Czwd+vPyJzkN6rqhqo6gcHg/USo7Nlme94M+EPgn1r7XUle0NrfPM13TnbcS4G3t8thJFnYbiL4JvDmJNu0sZff7+u3S9NxTEXqbpsk1zOYBXctcDZw0iT93pXktQzOYm4EvgK8Evg28HHgRQz+o39R6388cDGDNxT+ANhuiu/f4LhV9VCSFwDfbm9w/gXw76vq2iTnAdczGMP5X5vyw6WunKVYktQbL39JknpjqEiSemOoSJJ6Y6hIknpjqEiSemOoSJJ6Y6hIknpjqEiSevP/AQfaa5oYJqxDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(train.Disbursed);\n",
    "pyplot.xlabel('Disbursed');\n",
    "pyplot.ylabel('Number of occurrences');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "样本不均衡，使用交叉验证"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "默认参数，此时学习率为0.1，比较大，观察弱分类数目的大致范围\n",
    "（采用默认参数配置，看看模型是过拟合还是欠拟合）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = train['Disbursed']\n",
    "train = train.drop([\"Disbursed\"], axis=1)\n",
    "X_train = np.array(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 87020 entries, 0 to 87019\n",
      "Data columns (total 51 columns):\n",
      "Existing_EMI                     87020 non-null float64\n",
      "Loan_Amount_Applied              87020 non-null float64\n",
      "Loan_Tenure_Applied              87020 non-null float64\n",
      "Monthly_Income                   87020 non-null int64\n",
      "Var4                             87020 non-null int64\n",
      "Var5                             87020 non-null int64\n",
      "Age                              87020 non-null int64\n",
      "EMI_Loan_Submitted_Missing       87020 non-null int64\n",
      "Interest_Rate_Missing            87020 non-null int64\n",
      "Loan_Amount_Submitted_Missing    87020 non-null int64\n",
      "Loan_Tenure_Submitted_Missing    87020 non-null int64\n",
      "Processing_Fee_Missing           87020 non-null int64\n",
      "Device_Type_0                    87020 non-null int64\n",
      "Device_Type_1                    87020 non-null int64\n",
      "Filled_Form_0                    87020 non-null int64\n",
      "Filled_Form_1                    87020 non-null int64\n",
      "Gender_0                         87020 non-null int64\n",
      "Gender_1                         87020 non-null int64\n",
      "Var1_0                           87020 non-null int64\n",
      "Var1_1                           87020 non-null int64\n",
      "Var1_2                           87020 non-null int64\n",
      "Var1_3                           87020 non-null int64\n",
      "Var1_4                           87020 non-null int64\n",
      "Var1_5                           87020 non-null int64\n",
      "Var1_6                           87020 non-null int64\n",
      "Var1_7                           87020 non-null int64\n",
      "Var1_8                           87020 non-null int64\n",
      "Var1_9                           87020 non-null int64\n",
      "Var1_10                          87020 non-null int64\n",
      "Var1_11                          87020 non-null int64\n",
      "Var1_12                          87020 non-null int64\n",
      "Var1_13                          87020 non-null int64\n",
      "Var1_14                          87020 non-null int64\n",
      "Var1_15                          87020 non-null int64\n",
      "Var1_16                          87020 non-null int64\n",
      "Var1_17                          87020 non-null int64\n",
      "Var1_18                          87020 non-null int64\n",
      "Var2_0                           87020 non-null int64\n",
      "Var2_1                           87020 non-null int64\n",
      "Var2_2                           87020 non-null int64\n",
      "Var2_3                           87020 non-null int64\n",
      "Var2_4                           87020 non-null int64\n",
      "Var2_5                           87020 non-null int64\n",
      "Var2_6                           87020 non-null int64\n",
      "Mobile_Verified_0                87020 non-null int64\n",
      "Mobile_Verified_1                87020 non-null int64\n",
      "Source_0                         87020 non-null int64\n",
      "Source_1                         87020 non-null int64\n",
      "Source_2                         87020 non-null int64\n",
      "Source_3                         87020 non-null int64\n",
      "Source_4                         87020 non-null int64\n",
      "dtypes: float64(3), int64(48)\n",
      "memory usage: 33.9 MB\n"
     ]
    }
   ],
   "source": [
    "train.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def modelfit(alg, X_train, y_train, useTrainCV=True, cv_folds=None, early_stopping_rounds=100):\n",
    "    \n",
    "    if useTrainCV:\n",
    "        xgb_param = alg.get_xgb_params()\n",
    "        #xgb_param['num_class'] = 9\n",
    "        \n",
    "        xgtrain = xgb.DMatrix(X_train, label = y_train)\n",
    "        \n",
    "        cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], folds =cv_folds,\n",
    "                         metrics='logloss', early_stopping_rounds=early_stopping_rounds)\n",
    "        \n",
    "        n_estimators = cvresult.shape[0]\n",
    "        alg.set_params(n_estimators = n_estimators)\n",
    "        \n",
    "        print cvresult\n",
    "        #result = pd.DataFrame(cvresult)   #cv缺省返回结果为DataFrame\n",
    "        #result.to_csv('my_preds.csv', index_label = 'n_estimators')\n",
    "        cvresult.to_csv('my_preds4_2_3_699.csv', index_label = 'n_estimators')\n",
    "        \n",
    "        # plot\n",
    "        test_means = cvresult['test-logloss-mean']\n",
    "        test_stds = cvresult['test-logloss-std'] \n",
    "        \n",
    "        train_means = cvresult['train-logloss-mean']\n",
    "        train_stds = cvresult['train-logloss-std'] \n",
    "\n",
    "        x_axis = range(0, n_estimators)\n",
    "        pyplot.errorbar(x_axis, test_means, yerr=test_stds ,label='Test')\n",
    "        pyplot.errorbar(x_axis, train_means, yerr=train_stds ,label='Train')\n",
    "        pyplot.title(\"XGBoost n_estimators vs Log Loss\")\n",
    "        pyplot.xlabel( 'n_estimators' )\n",
    "        pyplot.ylabel( 'Log Loss' )\n",
    "        pyplot.savefig( 'n_estimators4_2_3_699.png' )\n",
    "    \n",
    "    #Fit the algorithm on the data\n",
    "    alg.fit(X_train, y_train, eval_metric='logloss')\n",
    "        \n",
    "    #Predict training set:\n",
    "    train_predprob = alg.predict_proba(X_train)\n",
    "    logloss = log_loss(y_train, train_predprob)\n",
    "\n",
    "        \n",
    "    #Print model report:\n",
    "    print (\"logloss of train :\" )\n",
    "    print logloss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     test-logloss-mean  test-logloss-std  train-logloss-mean  \\\n",
      "0             0.603504          0.000058            0.603484   \n",
      "1             0.530127          0.000040            0.530096   \n",
      "2             0.469059          0.000071            0.468993   \n",
      "3             0.417550          0.000084            0.417452   \n",
      "4             0.373641          0.000090            0.373531   \n",
      "5             0.335965          0.000082            0.335821   \n",
      "6             0.303403          0.000097            0.303231   \n",
      "7             0.275145          0.000114            0.274930   \n",
      "8             0.250492          0.000116            0.250243   \n",
      "9             0.228935          0.000124            0.228650   \n",
      "10            0.210002          0.000129            0.209675   \n",
      "11            0.193357          0.000146            0.192983   \n",
      "12            0.178674          0.000151            0.178267   \n",
      "13            0.165715          0.000179            0.165262   \n",
      "14            0.154256          0.000169            0.153769   \n",
      "15            0.144110          0.000179            0.143582   \n",
      "16            0.135125          0.000182            0.134554   \n",
      "17            0.127184          0.000208            0.126550   \n",
      "18            0.120103          0.000227            0.119413   \n",
      "19            0.113868          0.000252            0.113127   \n",
      "20            0.108325          0.000257            0.107524   \n",
      "21            0.103398          0.000278            0.102531   \n",
      "22            0.099028          0.000296            0.098099   \n",
      "23            0.095136          0.000309            0.094151   \n",
      "24            0.091691          0.000322            0.090665   \n",
      "25            0.088646          0.000360            0.087560   \n",
      "26            0.085951          0.000368            0.084772   \n",
      "27            0.083528          0.000391            0.082297   \n",
      "28            0.081397          0.000388            0.080102   \n",
      "29            0.079494          0.000392            0.078137   \n",
      "..                 ...               ...                 ...   \n",
      "98            0.063511          0.001266            0.058304   \n",
      "99            0.063486          0.001294            0.058221   \n",
      "100           0.063475          0.001284            0.058159   \n",
      "101           0.063462          0.001307            0.058103   \n",
      "102           0.063448          0.001311            0.058063   \n",
      "103           0.063461          0.001330            0.058010   \n",
      "104           0.063458          0.001353            0.057943   \n",
      "105           0.063454          0.001342            0.057875   \n",
      "106           0.063435          0.001332            0.057812   \n",
      "107           0.063447          0.001353            0.057732   \n",
      "108           0.063448          0.001371            0.057657   \n",
      "109           0.063444          0.001365            0.057599   \n",
      "110           0.063440          0.001369            0.057527   \n",
      "111           0.063438          0.001365            0.057477   \n",
      "112           0.063449          0.001384            0.057400   \n",
      "113           0.063447          0.001385            0.057350   \n",
      "114           0.063434          0.001379            0.057294   \n",
      "115           0.063429          0.001390            0.057210   \n",
      "116           0.063429          0.001389            0.057149   \n",
      "117           0.063433          0.001374            0.057105   \n",
      "118           0.063418          0.001345            0.057033   \n",
      "119           0.063410          0.001346            0.056973   \n",
      "120           0.063407          0.001355            0.056912   \n",
      "121           0.063408          0.001350            0.056823   \n",
      "122           0.063415          0.001333            0.056756   \n",
      "123           0.063423          0.001325            0.056677   \n",
      "124           0.063420          0.001322            0.056647   \n",
      "125           0.063410          0.001311            0.056576   \n",
      "126           0.063394          0.001294            0.056529   \n",
      "127           0.063391          0.001287            0.056501   \n",
      "\n",
      "     train-logloss-std  \n",
      "0             0.000040  \n",
      "1             0.000023  \n",
      "2             0.000030  \n",
      "3             0.000023  \n",
      "4             0.000036  \n",
      "5             0.000052  \n",
      "6             0.000082  \n",
      "7             0.000085  \n",
      "8             0.000062  \n",
      "9             0.000074  \n",
      "10            0.000091  \n",
      "11            0.000103  \n",
      "12            0.000117  \n",
      "13            0.000111  \n",
      "14            0.000126  \n",
      "15            0.000131  \n",
      "16            0.000144  \n",
      "17            0.000152  \n",
      "18            0.000166  \n",
      "19            0.000153  \n",
      "20            0.000163  \n",
      "21            0.000218  \n",
      "22            0.000225  \n",
      "23            0.000225  \n",
      "24            0.000254  \n",
      "25            0.000254  \n",
      "26            0.000276  \n",
      "27            0.000263  \n",
      "28            0.000269  \n",
      "29            0.000262  \n",
      "..                 ...  \n",
      "98            0.000142  \n",
      "99            0.000141  \n",
      "100           0.000178  \n",
      "101           0.000162  \n",
      "102           0.000152  \n",
      "103           0.000158  \n",
      "104           0.000167  \n",
      "105           0.000172  \n",
      "106           0.000187  \n",
      "107           0.000202  \n",
      "108           0.000201  \n",
      "109           0.000201  \n",
      "110           0.000206  \n",
      "111           0.000199  \n",
      "112           0.000239  \n",
      "113           0.000244  \n",
      "114           0.000238  \n",
      "115           0.000248  \n",
      "116           0.000237  \n",
      "117           0.000223  \n",
      "118           0.000261  \n",
      "119           0.000255  \n",
      "120           0.000235  \n",
      "121           0.000240  \n",
      "122           0.000221  \n",
      "123           0.000241  \n",
      "124           0.000240  \n",
      "125           0.000246  \n",
      "126           0.000228  \n",
      "127           0.000226  \n",
      "\n",
      "[128 rows x 4 columns]\n",
      "logloss of train :\n",
      "0.057589690082567264\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEXCAYAAABCjVgAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XmcHXWd7//X+5xekk6n01k6CdlICIEQFkEjizCIy8wAOoDIIIz4E4YZxvnJT9QZHbh6/f0ud5hR9LrMlZ8Oo4IbIOIWEUUUUFFZEmQNBELI0pCls3TSnaXXz/2jqjsnTXe6O+nT1Z3zfj4e9ehTVd9T9ek63ed9qurUtxQRmJmZAeSyLsDMzEYOh4KZmXVzKJiZWTeHgpmZdXMomJlZN4eCmZl1cyiYFZD03yR9Les6zLLiUBhlJFVLWi3pbwqmjZe0VtJFBdMWS7pb0jZJjZKWS7pB0sR0/uWSOiQ1p8MqSf9Y5NrPklRfzHUMRm/1RMS/RcTfFWl9qyW9vRjLLobher1G23Y51DkURpmIaAauAr4kqS6dfCOwNCLuApD0JuBB4PfAwoioBc4G2oHXFSzujxFRHRHVwEXAjZJOGp7fxAZDUlnWNViJiAgPo3AAbgVuB84CtgCHFcx7CPjf/Tz/cuChHtMeBf6mYPw84FmgkSRkjimYd0w6rTFtc17BvHOB5UAT8Arwz8A4YDfQCTSnw4w+fq+bgJ+lz38EmD+A7bEQuA/YCqwALj6QeoD/D/hO+ry5QABXAOuAbcAHgDcCT6W/+5cL1jMfuD99PTYD3wVq03nfTte1O13XxwewjVcD/5KuqwUoS8dfSX+XFcDbetkWpwIbgHzBtHcBT6WPTwaWAjuAjcDn+9imZwH1fcybAHwLaADWAJ8Ecum8PPC/0m3wMnB1uh3L+ljWauDtfcz7e2Bl+rou6fqbAQR8AdgEbE+30XF9vd5Z/7+OpiHzAjwc4AsHE4H16T/eFQXTxwEdwFn9PP9yCkIhfaNrBI5Kx48CdgJ/DpQDH0//OSvS8ZXAf0vH35r+Ax6dPnc98GcFdb4+fdznm0xBHbembwAnp2+C3wXu6Oc540jetK9In/P6dLscO9h66D0UvgqMAf4C2AP8GJgKzEzflN6ctj8y3V6VQB3wW+CLBcve581vf9u4oP0TwGxgLHB0+nvOKKiv18AEXgL+vGD8+8C16eM/Au9LH1cDp/axjD5fL5JA+AkwPq3jBeDKdN4HSN6UZ6Xb+1ccQCikf1eb09ezEvjfwG/TeX8JLANqSQLiGNIPRn293h4GNvjw0SgVEdtIPmFWAT8smDWR5LDghq4Jkm5MzyvslPTJgranptObSfYSvg28mM57D/CziLgvItqAz5G8Mb2J5JNoNfDpiGiNiPuBu4FL0+e2AYsk1UTEtoh4fJC/3g8j4tGIaCcJhRP7af9OYHVE3BIR7en6fkBySGwo6vmfEbEnIn5J8iZ+e0RsiohXgN8BJwFExMp0e7VERAPweeDN+1nu/rZxl/+IiHURsZsk7CvT36U8IlZHxEt9LPt20tdD0niST8+3F2yPIyVNiYjmiHh4MBtDUj6t/bqIaIqI1SR7Bu9Lm1wMfCki6tO/008PZvkF3gt8IyIej4gW4DrgNElz099hPMkeoiLiuYhYX/D7HczrXdIcCqOUpMtIPqH9CvhMwaxtJIcpDuuaEBEfj+S8wo9IPkl3eTgiaiM5pzAdOBb4t3TeDJLDAl3L6CT5lDoznbcundZlTToP4N0kb0JrJP1G0mmD/PU2FDzeRRJA+3M4cEoacI2SGkneUKYPUT0bCx7v7mW8GkDSVEl3SHpF0g7gO8CU/Sx3f9u4y7qC+SuBD5PszWxK1zWjj2XfBlwoqRK4EHg8IrrWdSXJXsrzkh6T9M791NibKSR7iGsKphW+/jMK6+7xeDB6bp9mkkNzM9MPIl8mOdS4UdLNkmrSpgf7epc0h8IoJGkqyfHUvwf+AbhY0pkAEbGT5Dj8hYNZZkRsJPl0/VfppFdJ3my71imSwxivpPNmSyr8+5mTziMiHouI80kOsfwYuLNrNYOpaRDWAb9JA65rqI6Ifxzmev49XeYJEVEDXEZyaKNLz/Xtbxv3+pyIuC0izkifF+z7gaCw3XKSN9RzgL8hCYmueS9GxKUk2+MzwF2Sxg3812Qzyafxwwumdb/+JIdvZhXMmz2IZRfquX3GAZPZ+3f2HxHxBpIPM0cBH0un9/V62wA4FEanLwM/jogH0l3mjwP/lX4qJB3/W0nXpgGCpFnAvL4WKGkyycnIZ9NJdwLvkPQ2SeXAP5Gc7PwDSejsBD4uqVzSWSRhcoekCknvlTQhPSSyg+SwBySfsCdLmjBE26HL3cBRkt6X1lMu6Y2SjhnmesaTnERulDST9E2qwEbgiILx/W3j15B0tKS3pq/zHpK9lI7e2qZuAz4EnElyTqFrOZdJqkv3TBrTyX0uR9KYwoFkT/RO4Ib069CHAx8l2TPq+r2ukTRTUi3JyfH+lPdYT1la/xWSTkx/538DHomI1enre0q63Xam26Ojn9fbBiLrkxoeBjcAF5B8gqrtMf3XwA0F46cA95D80zcCzwA3AJPT+ZeT/LN0ffNmE8kx56kFy3gXyQnD7cBvSE/cpvOOTadtT9u8K51eAfyC5DDWDuAx4IyC532D5BBAI31/++hfC8bPop+T02m7o0m+sdSQLv9+knMRg6qH3k80lxW0r6fgJD7JG+EnC7bJsnR7PkHyJl9f0PZ8YG26rn8ewDZezb4npk8gOffTRHIy/u7etmFB+zkkb+A/6zH9O+nr3UzyIeCCPp5/Vvr79xyOJDl39Z10e68DPsXebx+VkezJbiH59tFHSPYs1Md6Vveyjn9N532A5KR51+87K53+NpJvHDWz95te1f293h76H5RuYDOzopB0DvDViDi838aWOR8+MrMhJWmspHMllaWH0f5fki852CjgPQUbFST9GfDz3uZF8u0pGyEkVZEcCltIct7jZ8A1EbEj08JsQBwKZmbWzYePzMys26jrZGvKlCkxd+7crMswMxtVli1btjki6vprN+pCYe7cuSxdujTrMszMRhVJa/pv5cNHZmZWwKFgZmbdHApmZtbNoWBmZt0cCmZm1s2hYGZm3YoaCpLOlrRC0kpJ1/bR5mJJyyU9K+m23tqYmdnwKFoopLfsu4nkJh+LgEslLerRZgHJLfZOj4hjSe4qVRSrlj/Kw3feSHR29t/YzKxEFXNP4WRgZUSsiohW4A6S/uQL/T1wUyT3cSUiNhWrmE2P38Opy2+gace2Yq3CzGzUK2YozGTfe7PWs++9ZyG5hd5Rkn4v6WFJZxermLKaaQBsb6gv1irMzEa9YoaCepnWs0vWMmAByR2eLgW+lt6+b98FSVdJWippaUNDwwEVU1mb3MO9acv6A3q+mVkpKGYo1LPvDbtnkdxGsmebn0REW0S8DKwgCYl9RMTNEbE4IhbX1fXbn1OvqifPAGD3NoeCmVlfihkKjwELJM2TVAFcAizp0ebHwFsAJE0hOZy0qhjF1ExJQqF9+4ZiLN7M7JBQtFCIiHbgauBe4Dngzoh4VtL1ks5Lm90LbJG0HHgA+FhEbClGPbWTD6MjRGdz0c5lm5mNekXtOjsi7gHu6THtUwWPA/hoOhRVvqyMzZpAfteBnZMwMysFJXVF845cLeV7irIjYmZ2SCipUNhZPomqVoeCmVlfSioUWionM77DF6+ZmfWlpEKhvaqOiZ2N7urCzKwPJRUKVE9lrFppbmrMuhIzsxGppEIhPz7p6qKx4ZWMKzEzG5lKKhTGpF1dNG/peWG1mZlBiYVC1SR3dWFmtj8lFQoT6pJQaNu+MeNKzMxGppIKhYlTDqMzRGeTu7owM+tNSYVCvqycRo0nt8uhYGbWm5IKBYDtuUlU7NmcdRlmZiNSyYVCc/lEqlq3Zl2GmdmIVHKh4K4uzMz6VnKh0D62jtrORpJeu83MrFDJhQLj6qhSCzubt2ddiZnZiFNyoZCvSbu62FifcSVmZiNPyYVCpbu6MDPrU8mFwriuri4a3dWFmVlPJRcKE6bMBKC10V1dmJn1VHKhMHHqTNojR2eT9xTMzHoquVDIl5WxRRMpa3YomJn1VHKhANBYNoUxuzdkXYaZ2YhTkqGws3IaNW0NWZdhZjbilGQotI6bzuSOLb6q2cysh5IMBWpmUK3dNO1wH0hmZoVKMhTKa2cBsO3VlzOuxMxsZClqKEg6W9IKSSslXdvL/MslNUh6Ih3+rpj1dKmqmw3Ajk1rhmN1ZmajRlmxFiwpD9wE/DlQDzwmaUlELO/R9HsRcXWx6ujNhKlzAdi9xf0fmZkVKuaewsnAyohYFRGtwB3A+UVc34BNPmwOAB2Nr2RciZnZyFLMUJgJrCsYr0+n9fRuSU9JukvS7CLW061yTBVbqSHX7E7xzMwKFTMU1Mu0nt8B/SkwNyJOAH4FfLPXBUlXSVoqaWlDw9BcX7A1X+cL2MzMeihmKNQDhZ/8ZwH7fDSPiC0R0ZKO/hfwht4WFBE3R8TiiFhcV1c3JMU1V05lfMumIVmWmdmhopih8BiwQNI8SRXAJcCSwgaSDisYPQ94roj17KNl7DQmdm4ZrtWZmY0KRfv2UUS0S7oauBfIA9+IiGclXQ8sjYglwIcknQe0A1uBy4tVz2vqGz+DiVua2L2zmbHjqodrtWZmI1rRQgEgIu4B7ukx7VMFj68DritmDX3J1ybnvDevX83sI4/LogQzsxGnJK9oBhg7Ob2AbaMvYDMz61KyoVAz7XAAdm9Zm3ElZmYjR8mGwuTD5gLQ5gvYzMy6lWwojBtfyw6q0A5fwGZm1qVkQwFga24Klbt8AZuZWZeSDoUdFVOpbtmYdRlmZiNGSYfCnqrDmNzhq5rNzLqUdCh01sxhEjvY2dSYdSlmZiNCSYdC+ZR5AGxc90LGlZiZjQwlHQrV048AYPurL2VciZnZyFDSoTBl9lEAtGxalXElZmYjQ0mHwqS6meyKSmh0VxdmZlDioaBcjk35qVQ2r+u/sZlZCSjpUADYXjmDCXt8VbOZGTgU2FM9m6kdG4nOzqxLMTPLXMmHQtTOoVq7adzqi9jMzEo+FCrrkq+lNqz1tQpmZiUfCjXTjwSgacPKjCsxM8teyYfC1DnJtQqtm1/OuBIzs+yVfCiMr53MdsaR2+47sJmZlXwoADTkpzN2Z33WZZiZZc6hAOwYO5PaFl+rYGbmUABaq2czvXMTnR0dWZdiZpYphwKgSYdToXYaNrgPJDMrbQ4FYMnqcgC2rF2RcSVmZtlyKAAfvOhsAJpe9QVsZlbaHArAtDkLaI08nZsdCmZW2hwKQL6snPX5GYzZ7pvtmFlpK2ooSDpb0gpJKyVdu592F0kKSYuLWc/+bB07h4l7fAGbmZW2ooWCpDxwE3AOsAi4VNKiXtqNBz4EPFKsWgaiZcJ8ZnSsp72tNcsyzMwyVcw9hZOBlRGxKiJagTuA83tp9z+BG4E9RaylX/kpR1KhDtav8XkFMytdxQyFmUDhfS7r02ndJJ0EzI6Iu/e3IElXSVoqaWlDQ8PQVwqMn3UMAFvWPFuU5ZuZjQbFDAX1Mi26Z0o54AvAP/W3oIi4OSIWR8Tiurq6ISxxr+nzjgdg93pfq2BmpauYoVAPzC4YnwUUdjA0HjgOeFDSauBUYElWJ5trp0xPekvd6vsqmFnpKmYoPAYskDRPUgVwCbCka2ZEbI+IKRExNyLmAg8D50XE0iLW1DeJDeWzGde8OpPVm5mNBEULhYhoB64G7gWeA+6MiGclXS/pvGKt92DsGDeXqa3r+m9oZnaIKuuvgaT5QH1EtEg6CzgB+FZENPb33Ii4B7inx7RP9dH2rIEUXEwdE+cztfEXNG3fyvgJk7Iux8xs2A1kT+EHQIekI4GvA/OA24paVUYqpx0NwIaX/Q0kMytNAwmFzvRQ0LuAL0bER4DDiltWNibNSa6ta1z3XMaVmJllYyCh0CbpUuD9QNf1BOXFKyk7049YRGeI9k2+gM3MStNAQuEK4DTghoh4WdI84DvFLSsblWPGsSE3lYptL2ZdiplZJvo90RwRy0n6JkLSRGB8RHy62IVlZdPYI5i866WsyzAzy0S/ewqSHpRUI2kS8CRwi6TPF7+0bOyZeDSzOl5hz+5dWZdiZjbsBnL4aEJE7AAuBG6JiDcAby9uWdmpmHEcZeqk/sUnsy7FzGzYDSQUyiQdBlzM3hPNh6wp808CYOvLT2RciZnZ8BtIKFxPclXySxHxmKQjgEP2TOyMI46nNfJ0bHgm61LMzIbdQE40fx/4fsH4KuDdxSwqS2UVlawqm03VNveWamalZyAnmmdJ+pGkTZI2SvqBpFnDUVxWto1bwPQ9vl+zmZWegRw+uoWkd9MZJDfJ+Wk67ZDVNuUYprGF7Vs3ZV2KmdmwGkgo1EXELRHRng63AsW5080IUTU7ueFO/YplGVdiZja8BhIKmyVdJimfDpcBW4pdWJamL3gDAE1rnsq4EjOz4TWQUPhbkq+jbgDWAxeRdH1xyKqbMY8mqmDT8qxLMTMbVv2GQkSsjYjzIqIuIqZGxAUkF7IdspTLUV8+jwlN7hjPzErLgd557aNDWsUI1DThKGa1vkxnR2fWpZiZDZsDDQUNaRUjkA47nvHazaurn8+6FDOzYXOgoRBDWsUINHnBqQCsf/4PGVdiZjZ8+ryiWVITvb/5CxhbtIpGiNkL30BrlNG29vGsSzEzGzZ9hkJEjB/OQkaa8ooxvFh+BDXb3AeSmZWOAz18VBK21R7H4S0v0NHRkXUpZmbDwqGwH7mZr2e8drPuxaezLsXMbFg4FPaj7uhTANi04uGMKzEzGx4Ohf2YddRJ7I4KOl7xyWYzKw393k+hj28hbQeWAv+U3l/hkJQvK2dNxZFM8MlmMysR/YYC8HngVeA2kq+jXgJMB1YA3wDOKlZxI8GOicdx3Maf0N7WRll5edblmJkV1UAOH50dEf8ZEU0RsSMibgbOjYjvARP390RJZ0taIWmlpGt7mf8BSU9LekLSQ5IWHeDvUTT52a+nSi2sWeF7NpvZoW8godAp6WJJuXS4uGBen1c2S8oDNwHnAIuAS3t5078tIo6PiBOBG0n2SkaUaQtPA2DzCl/ZbGaHvoGEwnuB9wGb0uF9wGWSxgJX7+d5JwMrI2JVRLQCdwDnFzaIiB0Fo+MYgd1nzJx/PDsYR9Q/lnUpZmZF1+85hfRE8l/1Mfuh/Tx1JrCuYLweOKVnI0kfJOl1tQJ4a28LknQVcBXAnDlz+it5SCmXZ3XVcUxv/NOwrtfMLAv97ilImiXpR5I2Sdoo6QeSZg1g2b31pPqaPYGIuCki5gP/AnyytwVFxM0RsTgiFtfVDf+dQHdPP5m5Uc/mja8O+7rNzIbTQA4f3QIsAWaQfPr/aTqtP/XA7ILxWSTfYurLHcAFA1jusKtdeCYAa554IONKzMyKayChUBcRt0REezrcCgzk4/pjwAJJ8yRVkHyVdUlhA0kLCkbfAbw4wLqH1bwTzqA1ymhZ9fusSzEzK6qBXKewWdJlwO3p+KXAlv6eFBHtkq4G7gXywDci4llJ1wNLI2IJcLWktwNtwDbg/QfySxRbxZgqnq84iolblmVdiplZUQ0kFP4W+DLwBZJzAn8ArhjIwiPiHuCeHtM+VfD4mgFXmrHtdYs56ZXvsrO5iXHVJd2ruJkdwvo9fBQRayPivIioi4ipEXEBcOEw1DaiVB15OhXqYNWTv8u6FDOzojnQDvE+OqRVjAKHn5R8W3bHCw4FMzt0HWgo9PZ100NazcSprMnNoXrDo1mXYmZWNAcaCiPuyuPhsGnSGzhyz9Ps2bM761LMzIqiz1CQ1CRpRy9DE8k1CyWn8ui3Mk4tvPj4g1mXYmZWFH2GQkSMj4iaXobxETGQby0dco44+R10hNjx7C+zLsXMrCh857VBqJ4wmZcqjmbyRl/EZmaHJofCIDVOP50FbS+wbUtD1qWYmQ05h8Ig1R7/F+QVrHz0nv4bm5mNMg6FQTrixLPYRSXtL96fdSlmZkPOoTBIZRVjWFl1IjO3PUJESX4z18wOYQ6FA9Ay583MifWsXfV81qWYmQ0ph8IBmPXG5EZ09Y/8OONKzMyGlkPhABw2/wTW5WZSvdrXK5jZocWhcIA2TH8Lx7Q8ybatm7MuxcxsyDgUDtDE119AhTpY8dCPsi7FzGzIOBQO0PyT3sJWasi94OsVzOzQ4VA4QMqX8fKkM1jY9DB79uzJuhwzsyHhUDgIFcf+FTXaxXMP/zzrUszMhoRD4SAcddo72RPl7HpqSdalmJkNCYfCQaisqmFFzWkcvfXXtLS2ZF2OmdlBcygcpPwJf80UtvPM73+WdSlmZgfNoXCQjv6zd9PMWFr/9L2sSzEzO2gOhYNUPmYcL0w8i2O3/5adO5uzLsfM7KA4FIbAuMXvoUa7eObBH2RdipnZQXEoDIEFp7yTbdSgZ+/KuhQzs4PiUBgCubJyfpU7nRN2/pGGho1Zl2NmdsCKGgqSzpa0QtJKSdf2Mv+jkpZLekrSryUdXsx6iunUi65hjNp47pdfz7oUM7MDVrRQkJQHbgLOARYBl0pa1KPZn4DFEXECcBdwY7HqKbbZi07jpfIFzFx5B50dnVmXY2Z2QIq5p3AysDIiVkVEK3AHcH5hg4h4ICJ2paMPA7OKWE/RNR37XubHGp561PdvNrPRqZihMBNYVzBen07ry5VAr50ISbpK0lJJSxsaGoawxKG18O1XsItKdv3xa1mXYmZ2QIoZCuplWq93upd0GbAY+Gxv8yPi5ohYHBGL6+rqhrDEoTWmupbnp/wlJ26/n82bR254mZn1pZihUA/MLhifBbzas5GktwOfAM6LiFHfgVDdm/+BKrXw7a9+OutSzMwGrZih8BiwQNI8SRXAJcA+3YlKOgn4T5JA2FTEWobN7ONOZ2XFQv66/afsaWnNuhwzs0EpWihERDtwNXAv8BxwZ0Q8K+l6SeelzT4LVAPfl/SEpNHfB7VE2ylXM4uNLLv3O1lXY2Y2KIro9TD/iLV48eJYunRp1mXsV3S0s+GGRWzVRBZ98mGk3k6vmJkNH0nLImJxf+18RXMRKF/GxkVXcmzH8zz+h19mXY6Z2YA5FIpk0bn/yA7G0f67L2VdipnZgDkUiqSiqoaX5v0Np+z5PU8ufSjrcszMBsShUETHvOs6mqii5b5/ZbSduzGz0uRQKKIxNZN5af77Obnlj/zpkQezLsfMrF8OhSI75sJ/YQfVdN5/g/cWzGzEcygUWeW4iXyv4l0sbn2MRx78WdblmJntl0NhGFzxkX+nQVOY+Nv/zp7WtqzLMTPrk0NhGJSNHc+20z/B0bGK39/lr6ia2cjlUBgmR73tClaOOY7XrfgPNmw6JLp5MrNDkENhuEhUX/A5JrGDZ2+7LutqzMx65VAYRtMXnsazMy7krG0/4I+/vTfrcszMXsOhMMwWvu/zbM1Npu7+f6axqTnrcszM9uFQGGblVbXs/IvPciRreeSbn8i6HDOzfTgUMjD3tAtZnl/IWxu+zUMP/iLrcszMujkUMrLgIz9nW34ycx/4f1j36oasyzEzAxwKmSmvnkS8++tM12bW3HolrW0dWZdkZuZQyNK0Y8/kpeM+zBmtD/Glf/+Y+0Yys8w5FDJ29IX/nRdrT+cjHbfwy7vvzLocMytxDoWs5XLM/4fb2Vgxh1OWfoRHlz6WdUVmVsIcCiNAbuwEJv7dD1Euz7SfXsYzL7yYdUlmVqIcCiNE1bT5dLznNqZqG2W3XcSLa+qzLsnMSpBDYQSZtPDPaDrvFuZTz85b381Lr2zMuiQzKzEOhRFm6uvfQcNffJnj4wWa/+udLF+1NuuSzKyEOBRGoBlvupTNZ3+VRaxC3/orHn/O5xjMbHg4FEaoaae+hx0XfJt58QqT7ziXex/8bdYlmVkJcCiMYJNPPJf2991Nbb6V0x54D7fd/k3aOzqzLsvMDmFFDQVJZ0taIWmlpGt7mX+mpMcltUu6qJi1jFbVR55K1Qd/w67cON7z/DV8/wvX8Oq2nVmXZWaHqKKFgqQ8cBNwDrAIuFTSoh7N1gKXA7cVq45DQfnkuUy/9k+8OvtcLm3+Fqu/dA73Pfp01mWZ2SGomHsKJwMrI2JVRLQCdwDnFzaIiNUR8RTgYyL9qRzP7Cu/y+a3fJbFLGfxz87m61+5kYYde7KuzMwOIcUMhZnAuoLx+nTaoEm6StJSSUsbGhqGpLhRSWLKm68i94Hf0VIzlys33sATnzuXO+97iDafazCzIVDMUFAv0w6oG9CIuDkiFkfE4rq6uoMsa/Qrm34M0z/8Gzaf9knOzD/DeQ9dwG03/t/86slV7mnVzA5KMUOhHphdMD4LeLWI6yst+TKm/OXHqLhmGdvm/Dnvb7md1/3wLG7+3LU88Ow6OjsdDmY2eMUMhceABZLmSaoALgGWFHF9JUm1sznsyttpv/wXxOQF/MPOr3LsnWfwrc98kB///il2tbZnXaKZjSIq5uEGSecCXwTywDci4gZJ1wNLI2KJpDcCPwImAnuADRFx7P6WuXjx4li6dGnRah7VImhf+QCb7/tfTN/0EHuinCW8mfqjL+esM87gpNm1SL0d1TOzQ52kZRGxuN92o+0YtENhYGLTczTc90Umrvwh5dHKY51H8ceqtzLupIs488SFHDm12gFhVkIcCpbYuZmWR29l97LbqW1eSXvk+F3n8fxh7FlUHHM2p59wFIsPn0RFmS9uNzuUORRsXxGw8Vmal96Olt3KuGimM8RTMY/fdZ7AlulnUnvUaZx8xFROmjORsRX5rCs2syHkULC+dXbCq4/TuuI+di6/lwlbniRHJ80xhic75/MkC9hYczwx4w3Mnn04i2bUsHD6eCaNq/AhJ7NRyqFgA7d7G6z6Da0v/YY9qx+leutz5OgAoD6m8HznbJ6POawtm8euiQsZM+1IZk+p5fDJVRw2YQwzascytaaSyjLvXZiNVA4FO3Ctu2D9E1D/GC3rnqB9/dOM3fEyuUi+3tpBjvUxibWdU1kbU1kb01gbU9kxdhZRM5Oq2mlMq62LrV55AAALlklEQVSitqqCSVXlTBxXQW1VBROryplYVUFtVTnVlWXe6zAbRgMNhbLhKMZGmYoqOPxNcPibqAQqAdpbYPMLsHE5+a0vMX3Ly0xueInF25+hYs+DyfM6gG3QsS3HFiawobOWTZEMq5lIQ0xge4yjkWqac9XEmIl0jqlFlTVUVVYwrjLPuMqyZKhIHldXllFVUUZlWY7K8hwV+RwVZTkqy/Lpz65h73hFWY58TuRzoiwnh4/ZIDgUbGDKKmH68clA8ofT/cfT0gyNa2Dry7DjVfLNG5jatJG6pg107FgPzU+T370F9ezlpB1ohs7mHM2qpknj2E41TTGWps4Kmjor2R2V7GQMDVSyK8awK/25kzHsppKdUcluxrCTpG0rZbRSTivltJGnq7cVAZXlOcpyOXKCsnwaHErDI7/3cWGg5NKfe6fnkunqmg9SMi4gp2RcomBa0g6Uzk+npWGVPN47HYG66tbe/mLUx3S093fsyj+hgscFzy8IyH7bFqxr73N6W1fB9AOti70T907XPm36q+s1bXtZ12AM/rPE4Ncy2HUsOqyG2ZOqBr2ewXAo2MGrrIZpxyZDAVHwB9bRBru2wO5G2NOYnMdIh9zubdTsbqRm9zZm7t4Grc3QupNo3UK07oTWneTaBn8PiUB05CpoVzkduXLaVZEO5bSrnDZV0K4y2lRBG+W0qZz2KKO9PU8beTrI0UYZ7eRoj2Rae+Roizxt5JLHlNEeOdrTee3K0xZ5tu7upC3ydJCnnRxVlRU0tnTSETk6yNFJjorycprb9k7rQHSSoyNy5PJ5WtrV3bYDEbk8bZ2F05IhfK+skjF3chUPfuwtRV2HQ8GGR74cxk9PhgESBZ+9OjuhfXdyvqO1Gdp2QevONEDSx227oKM1OdTV0YLaWyhrb6Gse1ortO8peNzS3Zb25uRnRxt0tidDRxt0dqTj6fQYYG+0Pc+5d9D7f1v5fpYxmP9O5YlcHpSHXB6US38WPs51zw/l012TPJDrbhPS3ucoR6Q/Ub7gcc9pSpeX26dN9zq62rF3fuQKx/cuO9i7vL3z1D0vuupEe+d3P06f3z0/Wcbe36mXOskl2y0NXQrq3Kc2lC43B7nC5yXLD+X3XVbXtk63T2/iAPoHPWzC2EE/Z7AcCjY65HJQMS4ZyLCn3M7OvaHRmYZGYZDsEyjtaaik7aKj4Gdnj/GOJHC6fvY5r+9lqK9lFy63q+0+6yoYCtcfncn1LZ0d0NlaMK2rXfQyreu50cfyOtP6ek7r4AA7UR4FtDdIC8NZacAUhOzeNuplWg7O/Bgcd2FRq3UomA1GLge5CqAi60oOPV1B8pqQKQia10wrbBe9TOsRbr2FUX/hNuh19vbcIQjUzg4YU1P0l8GhYGYjQ9dhntcce7Ph5DNUZmbWzaFgZmbdHApmZtbNoWBmZt0cCmZm1s2hYGZm3RwKZmbWzaFgZmbdRt39FCQ1AGsO8OlTgM1DWM5wc/3Zcv3Zcv0H5/CI6LePmFEXCgdD0tKB3GRipHL92XL92XL9w8OHj8zMrJtDwczMupVaKNycdQEHyfVny/Vny/UPg5I6p2BmZvtXansKZma2Hw4FMzPrVjKhIOlsSSskrZR0bdb19EfSbEkPSHpO0rOSrkmnT5J0n6QX058Ts661L5Lykv4k6e50fJ6kR9LavydpRN++TFKtpLskPZ++DqeNlu0v6SPp380zkm6XNGakb39J35C0SdIzBdN63d5K/Ef6//yUpNdnV3l3rb3V/9n07+cpST+SVFsw77q0/hWS/jKbql+rJEJBUh64CTgHWARcKmlRtlX1qx34p4g4BjgV+GBa87XAryNiAfDrdHykugZ4rmD8M8AX0tq3AVdmUtXAfQn4RUQsBF5H8ruM+O0vaSbwIWBxRBxHciuzSxj52/9W4Owe0/ra3ucAC9LhKuArw1Tj/tzKa+u/DzguIk4AXgCuA0j/ly8Bjk2f8/+n71OZK4lQAE4GVkbEqohoBe4Azs+4pv2KiPUR8Xj6uInkDWkmSd3fTJt9E7ggmwr3T9Is4B3A19JxAW8F7kqbjNjaASTVAGcCXweIiNaIaGSUbH+SW+2OlVQGVAHrGeHbPyJ+C2ztMbmv7X0+8K1IPAzUSjpseCrtXW/1R8QvI6I9HX0YmJU+Ph+4IyJaIuJlYCXJ+1TmSiUUZgLrCsbr02mjgqS5wEnAI8C0iFgPSXAAU7OrbL++CHwc6EzHJwONBf8gI/01OAJoAG5JD4F9TdI4RsH2j4hXgM8Ba0nCYDuwjNG1/bv0tb1H4//03wI/Tx+P2PpLJRTUy7RR8V1cSdXAD4APR8SOrOsZCEnvBDZFxLLCyb00HcmvQRnweuArEXESsJMReKioN+lx9/OBecAMYBzJ4ZaeRvL278+o+nuS9AmSQ8Lf7ZrUS7MRUX+phEI9MLtgfBbwaka1DJikcpJA+G5E/DCdvLFrNzn9uSmr+vbjdOA8SatJDtW9lWTPoTY9nAEj/zWoB+oj4pF0/C6SkBgN2//twMsR0RARbcAPgTcxurZ/l76296j5n5b0fuCdwHtj74VhI7b+UgmFx4AF6bcvKkhO8CzJuKb9So/Bfx14LiI+XzBrCfD+9PH7gZ8Md239iYjrImJWRMwl2db3R8R7gQeAi9JmI7L2LhGxAVgn6eh00tuA5YyC7U9y2OhUSVXp31FX7aNm+xfoa3svAf6v9FtIpwLbuw4zjSSSzgb+BTgvInYVzFoCXCKpUtI8khPmj2ZR42tEREkMwLkkZ/9fAj6RdT0DqPcMkt3Jp4An0uFckmPzvwZeTH9OyrrWfn6Ps4C708dHkPzhrwS+D1RmXV8/tZ8ILE1fgx8DE0fL9gf+B/A88AzwbaBypG9/4HaScyBtJJ+kr+xre5Mcfrkp/X9+muSbViOx/pUk5w66/oe/WtD+E2n9K4Bzsq6/a3A3F2Zm1q1UDh+ZmdkAOBTMzKybQ8HMzLo5FMzMrJtDwczMujkUzMysm0PBbAAknSjp3ILx84aqC3ZJH5ZUNRTLMjtYvk7BbAAkXU5ygdTVRVj26nTZmwfxnHxEdAx1LWbeU7BDiqS56Q1x/iu9ycwvJY3to+18Sb+QtEzS7yQtTKf/dXpzmicl/TbtGuV64D2SnpD0HkmXS/py2v5WSV9RclOkVZLenN5w5TlJtxas7yuSlqZ1/Y902odIOq17QNID6bRLJT2d1vCZguc3S7pe0iPAaZI+LWl5egOXzxVni1rJyfqSag8ehnIA5pL0RnliOn4ncFkfbX8NLEgfn0LSRxMk3SbMTB/Xpj8vB75c8NzucZKbq9xB0vXC+cAO4HiSD13LCmrp6qIhDzwInJCOrwampI9nkPRdVEfSU+v9wAXpvAAu7loWSfcIKqzTg4eDHbynYIeilyPiifTxMpKg2EfaJfmbgO9LegL4T6DrJi2/B26V9Pckb+AD8dOICJJA2RgRT0dEJ/BswfovlvQ48CeSO271dve/NwIPRtLDaVdXy2em8zpIes2FJHj2AF+TdCGw6zVLMjsAZf03MRt1WgoedwC9HT7Kkdx05sSeMyLiA5JOIblz3BOSXtNmP+vs7LH+TqAs7Qnzn4E3RsS29LDSmF6W01s/+132RHoeISLaJZ1M0gPqJcDVJF2Umx0U7ylYSYrkhkUvS/pr6L4R/OvSx/Mj4pGI+BSwmaTf+yZg/EGssobkRj3bJU1j35veFC77EeDNkqak9+y9FPhNz4WlezoTIuIe4MMkPbqaHTTvKVgpey/wFUmfBMpJzgs8CXxW0gKST+2/TqetBa5NDzX9+2BXFBFPSvoTyeGkVSSHqLrcDPxc0vqIeIuk60jufSDgnojo7b4H44GfSBqTtvvIYGsy642/kmpmZt18+MjMzLr58JEd8iTdRHLf6EJfiohbsqjHbCTz4SMzM+vmw0dmZtbNoWBmZt0cCmZm1s2hYGZm3f4PyY3rcc0+MdwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "xgb7_1 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=1000,  #数值大没关系，cv会自动返回合适的n_estimators\n",
    "        max_depth=5,\n",
    "        min_child_weight=2,\n",
    "        gamma=0,\n",
    "        subsample=0.8,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel=0.7,\n",
    "        objective= 'binary:logistic',\n",
    "        reg_alpha=0.1,\n",
    "        reg_lambda=0.5,\n",
    "        seed=3)\n",
    "\n",
    "modelfit(xgb7_1, X_train, y_train, cv_folds = kfold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     test-logloss-mean  test-logloss-std  train-logloss-mean  \\\n",
      "0             0.683759          0.000006            0.683757   \n",
      "1             0.674558          0.000006            0.674555   \n",
      "2             0.665540          0.000010            0.665534   \n",
      "3             0.656698          0.000011            0.656689   \n",
      "4             0.648022          0.000013            0.648012   \n",
      "5             0.639513          0.000012            0.639502   \n",
      "6             0.631165          0.000014            0.631152   \n",
      "7             0.622977          0.000017            0.622960   \n",
      "8             0.614938          0.000017            0.614920   \n",
      "9             0.607049          0.000017            0.607029   \n",
      "10            0.599305          0.000018            0.599282   \n",
      "11            0.591700          0.000020            0.591676   \n",
      "12            0.584232          0.000020            0.584206   \n",
      "13            0.576897          0.000024            0.576870   \n",
      "14            0.569695          0.000029            0.569666   \n",
      "15            0.562617          0.000029            0.562586   \n",
      "16            0.555661          0.000034            0.555628   \n",
      "17            0.548828          0.000032            0.548795   \n",
      "18            0.542114          0.000034            0.542080   \n",
      "19            0.535516          0.000034            0.535479   \n",
      "20            0.529033          0.000036            0.528992   \n",
      "21            0.522656          0.000038            0.522614   \n",
      "22            0.516388          0.000038            0.516343   \n",
      "23            0.510224          0.000037            0.510176   \n",
      "24            0.504162          0.000038            0.504111   \n",
      "25            0.498201          0.000038            0.498148   \n",
      "26            0.492340          0.000041            0.492285   \n",
      "27            0.486573          0.000045            0.486516   \n",
      "28            0.480899          0.000045            0.480840   \n",
      "29            0.475318          0.000046            0.475256   \n",
      "..                 ...               ...                 ...   \n",
      "970           0.063452          0.001130            0.058457   \n",
      "971           0.063452          0.001129            0.058450   \n",
      "972           0.063451          0.001128            0.058443   \n",
      "973           0.063450          0.001128            0.058437   \n",
      "974           0.063450          0.001130            0.058431   \n",
      "975           0.063448          0.001130            0.058423   \n",
      "976           0.063449          0.001133            0.058416   \n",
      "977           0.063447          0.001132            0.058411   \n",
      "978           0.063448          0.001132            0.058406   \n",
      "979           0.063447          0.001132            0.058396   \n",
      "980           0.063445          0.001131            0.058390   \n",
      "981           0.063444          0.001131            0.058384   \n",
      "982           0.063443          0.001132            0.058377   \n",
      "983           0.063444          0.001133            0.058372   \n",
      "984           0.063442          0.001133            0.058367   \n",
      "985           0.063442          0.001133            0.058360   \n",
      "986           0.063441          0.001133            0.058355   \n",
      "987           0.063441          0.001135            0.058350   \n",
      "988           0.063440          0.001134            0.058345   \n",
      "989           0.063439          0.001134            0.058338   \n",
      "990           0.063440          0.001136            0.058334   \n",
      "991           0.063438          0.001136            0.058328   \n",
      "992           0.063437          0.001137            0.058322   \n",
      "993           0.063438          0.001138            0.058319   \n",
      "994           0.063437          0.001138            0.058311   \n",
      "995           0.063436          0.001138            0.058302   \n",
      "996           0.063432          0.001137            0.058293   \n",
      "997           0.063431          0.001136            0.058284   \n",
      "998           0.063429          0.001134            0.058277   \n",
      "999           0.063429          0.001135            0.058272   \n",
      "\n",
      "     train-logloss-std  \n",
      "0             0.000005  \n",
      "1             0.000004  \n",
      "2             0.000004  \n",
      "3             0.000004  \n",
      "4             0.000006  \n",
      "5             0.000006  \n",
      "6             0.000010  \n",
      "7             0.000008  \n",
      "8             0.000005  \n",
      "9             0.000006  \n",
      "10            0.000008  \n",
      "11            0.000006  \n",
      "12            0.000010  \n",
      "13            0.000007  \n",
      "14            0.000006  \n",
      "15            0.000007  \n",
      "16            0.000007  \n",
      "17            0.000007  \n",
      "18            0.000006  \n",
      "19            0.000005  \n",
      "20            0.000003  \n",
      "21            0.000005  \n",
      "22            0.000008  \n",
      "23            0.000014  \n",
      "24            0.000014  \n",
      "25            0.000015  \n",
      "26            0.000013  \n",
      "27            0.000013  \n",
      "28            0.000016  \n",
      "29            0.000016  \n",
      "..                 ...  \n",
      "970           0.000245  \n",
      "971           0.000242  \n",
      "972           0.000243  \n",
      "973           0.000240  \n",
      "974           0.000242  \n",
      "975           0.000243  \n",
      "976           0.000245  \n",
      "977           0.000243  \n",
      "978           0.000242  \n",
      "979           0.000242  \n",
      "980           0.000243  \n",
      "981           0.000244  \n",
      "982           0.000244  \n",
      "983           0.000244  \n",
      "984           0.000242  \n",
      "985           0.000241  \n",
      "986           0.000241  \n",
      "987           0.000242  \n",
      "988           0.000242  \n",
      "989           0.000244  \n",
      "990           0.000245  \n",
      "991           0.000244  \n",
      "992           0.000244  \n",
      "993           0.000245  \n",
      "994           0.000244  \n",
      "995           0.000247  \n",
      "996           0.000249  \n",
      "997           0.000249  \n",
      "998           0.000251  \n",
      "999           0.000253  \n",
      "\n",
      "[1000 rows x 4 columns]\n",
      "logloss of train :\n",
      "0.05893929050405004\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEXCAYAAABCjVgAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XmcHWWd7/HP95zeks5K0tmXDiEsYZeWxQ1ERgHHoI4iKF5RR2TmMrg7cHWckXF3xp3riA7gVRFRFBFRVMR1ABMwLAEC2dNZOyHp7Ekvv/tHVTenm+50d9Knq7vP9/161atPPc9TVb+qSs7v1PaUIgIzMzOAXNYBmJnZ4OGkYGZm7ZwUzMysnZOCmZm1c1IwM7N2TgpmZtbOScGsgKT/I+lbWcdhlhUnhSFG0ihJqyS9uaBstKQ1kt5QUFYn6S5J2yRtl/SEpE9KGp/WXy6pRdKudFgh6R+KHPs5kuqLuYy+6CqeiPhURPx9kZa3StJ5xZh3MQzU/hpq22W4c1IYYiJiF3AF8GVJNWnx54BFEfEjAEkvAn4H/Bk4NiLGAecDzcDJBbO7PyJGRcQo4A3A5ySdOjBrYn0hqSzrGKxERISHITgANwPfB84BtgJTC+r+BHy1h+kvB/7UqewvwJsLxhcAS4DtJEnmuIK649Ky7WmbBQV1FwJPADuBdcAHgWpgL9AK7EqHad2s1/XAz9PpHwTm9mJ7HAv8GngWWApcfCjxAP8GfDedrhYI4O3AWmAbcCXwQuDRdN2/VrCcucBv0/2xBfgeMC6t+066rL3psj7ci228CvjndFn7gbJ0fF26LkuBV3SxLc4ENgL5grLXAY+mn08HFgE7gE3AF7rZpucA9d3UjQX+H9AArAY+CuTSujzwn+k2WAlclW7Hsm7mtQo4r5u6dwHL0v16Z9u/GUDAF4HNQGO6jU7obn9n/f91KA2ZB+DhEHccjAc2pP/x3l5QXg20AOf0MP3lFCSF9ItuO3B0On40sBv4G6Ac+HD6n7MiHV8G/J90/Nz0P+Ax6bQbgJcWxPmC9HO3XzIFcdycfgGcnn4Jfg+4tYdpqkm+tN+eTvOCdLsc39d46Dop/BdQBbwS2AfcAUwCpqdfSmen7Y9Kt1clUAP8AfhSwbw7fPkdbBsXtF8MzARGAMek6zmtIL4uEyawHPibgvEfAtekn+8H3pp+HgWc2c08ut1fJAnhp8DoNI6ngXemdVeSfCnPSLf3bziEpJD+u9qS7s9K4KvAH9K6VwEPAeNIEsRxpD+MutvfHno3+PTREBUR20h+YY4EflxQNZ7ktODGtgJJn0uvK+yW9NGCtmem5btIjhK+AzyT1r0J+HlE/DoimoD/IPliehHJL9FRwGci4kBE/Ba4C7g0nbYJmC9pTERsi4iH+7h6P46Iv0REM0lSOKWH9n8LrIqImyKiOV3e7SSnxPojnn+PiH0R8SuSL/HvR8TmiFgH/BE4FSAilqXba39ENABfAM4+yHwPto3bfCUi1kbEXpJkX5muS3lErIqI5d3M+/uk+0PSaJJfz98v2B5HSZoYEbsi4oG+bAxJ+TT2ayNiZ0SsIjkyeGva5GLgyxFRn/47/Uxf5l/gLcCNEfFwROwHrgXOklSbrsNokiNERcSTEbGhYP0OZ3+XNCeFIUrSZSS/0H4DfLagahvJaYqpbQUR8eFIriv8hOSXdJsHImJcJNcUpgDHA59K66aRnBZom0crya/U6Wnd2rSszeq0DuDvSL6EVkv6vaSz+rh6Gws+7yFJQAczGzgjTXDbJW0n+UKZ0k/xbCr4vLeL8VEAkiZJulXSOkk7gO8CEw8y34Nt4zZrC+qXAe8lOZrZnC5rWjfzvgV4vaRK4PXAwxHRtqx3khylPCVpoaS/PUiMXZlIcoS4uqCscP9PK4y70+e+6Lx9dpGcmpue/hD5Gsmpxk2SbpA0Jm16uPu7pDkpDEGSJpGcT30X8G7gYkkvA4iI3STn4V/fl3lGxCaSX9evSYvWk3zZti1TJKcx1qV1MyUV/vuZldYREQsj4iKSUyx3ALe1LaYvMfXBWuD3aYJrG0ZFxD8McDyfTud5UkSMAS4jObXRpvPyDraNu5wmIm6JiJek0wUdfxAUtnuC5Av1AuDNJEmire6ZiLiUZHt8FviRpOreryZbSH6Nzy4oa9//JKdvZhTUzezDvAt13j7VwASe+3f2lYg4jeTHzNHAh9Ly7va39YKTwtD0NeCOiLgvPWT+MPDN9Fch6fg7JF2TJhAkzQDmdDdDSRNILkYuSYtuA14t6RWSyoEPkFzs/B+SpLMb+LCkcknnkCSTWyVVSHqLpLHpKZEdJKc9IPmFPUHS2H7aDm3uAo6W9NY0nnJJL5R03ADHM5rkIvJ2SdNJv6QKbAKOLBg/2DZ+HknHSDo33c/7SI5SWrpqm7oFuBp4Gck1hbb5XCapJj0y2Z4WdzsfSVWFA8mR6G3AJ9PboWcD7yc5Mmpbr/dImi5pHMnF8Z6Ud1pOWRr/2yWdkq7zp4AHI2JVun/PSLfb7nR7tPSwv603sr6o4aFvA/Bakl9Q4zqV3wt8smD8DOBukv/024HHgU8CE9L6y0n+s7TdebOZ5JzzpIJ5vI7kgmEj8HvSC7dp3fFpWWPa5nVpeQXwS5LTWDuAhcBLCqa7keQUwHa6v/voEwXj59DDxem03TEkdyw1pPP/Lcm1iD7FQ9cXmssK2tdTcBGf5IvwowXb5KF0ey4m+ZKvL2h7EbAmXdYHe7GNV9HxwvRJJNd+dpJcjL+rq21Y0H4WyRf4zzuVfzfd37tIfgS8tpvpz0nXv/NwFMm1q++m23st8DGeu/uojORIdivJ3UfvIzmyUDfLWdXFMj6R1l1JctG8bX1npOWvILnjaBfP3ek1qqf97aHnQekGNjMrCkkXAP8VEbN7bGyZ8+kjM+tXkkZIulBSWXoa7V9JbnKwIcBHCjYkSHop8Iuu6iK5e8oGCUkjSU6FHUty3ePnwHsiYkemgVmvOCmYmVk7nz4yM7N2Q66TrYkTJ0ZtbW3WYZiZDSkPPfTQloio6andkEsKtbW1LFq0KOswzMyGFEmre25V5NNHks6XtFTSMknXdFH/RUmL0+HptHsCMzPLSNGOFNJOs64n6QGyHlgo6c5IHr8HICLeV9D+n0g7FjMzs2wU80jhdGBZRKyIiAPArSRPdHbnUp7rxdHMzDJQzKQwnY69I9bTsffHdmnfKXNIuiboqv4KSYskLWpoaOj3QM3MLFHMpKAuyrp7KOIS4EcR0WXHVRFxQ0TURURdTU2PF8/NzOwQFTMp1NOxy9wZJB25deUSfOrIzCxzxUwKC4F5kuZIqiD54r+zcyNJx5D0uHh/EWMxM7NeKFpSiORVilcB9wBPArdFxBJJ10laUND0UpJ38Ba1v43lj/4PD9zyCaK1tefGZmYlqqgPr0XE3SR9+heWfazT+L8VM4Y2DY/9mjOf+QKNjVcydvzB3pBoZla6Sqbvo/LxydsBn13f3XvOzcysZJJC9cRZAOzYvCbjSMzMBq+SSQrjptQCsH/r2oM3NDMrYSWTFCZMmUVriJbGdVmHYmY2aJVMUiivqGSrxpHf2d2jEmZmVjJJAWBbWQ1V+zZlHYaZ2aBVUklhV+Vkxh7YnHUYZmaDVkklhQMjpzChZUvWYZiZDVollRQYM41R2svOxmezjsTMbFAqqaRQ1v4A28qMIzEzG5xKKimMbH+ArVevKjUzKzkllRTGTZkDwF4/wGZm1qWSSgoTp84GoGV7fcaRmJkNTiWVFCoqq9iCH2AzM+tOSSUFgG35iVTt9QNsZmZdKbmksKtyEmMOOCmYmXWl5JLCgeppTGxp8BvYzMy6UHJJIcbOZJT2smP71qxDMTMbdEouKVRMrAWgYe3T2QZiZjYIlVxSGD1lLgA7N/mpZjOzzkouKUycfhQA+7c4KZiZdVZySWHchMnsiUrY7qeazcw6K7mkoFyOzflJVO7yU81mZp0VNSlIOl/SUknLJF3TTZuLJT0haYmkW4oZT5vGyqmM3r9xIBZlZjaklBVrxpLywPXA3wD1wEJJd0bEEwVt5gHXAi+OiG2SJhUrnkL7qmdQu3fJQCzKzGxIKeaRwunAsohYEREHgFuBizq1eRdwfURsA4iIAXlXZoyZwVh2+2U7ZmadFDMpTAcKr+bWp2WFjgaOlvRnSQ9IOr+rGUm6QtIiSYsaGhoOO7Dy9FmFLfXPHPa8zMyGk2ImBXVRFp3Gy4B5wDnApcC3JI173kQRN0REXUTU1dTUHHZgoycfCUDjRt+WamZWqJhJoR6YWTA+A+jcZ3U98NOIaIqIlcBSkiRRVEe0PavQ4KRgZlaomElhITBP0hxJFcAlwJ2d2twBvBxA0kSS00krihgTABMmTWdflBN+VsHMrIOiJYWIaAauAu4BngRui4glkq6TtCBtdg+wVdITwH3AhyKi6D3VtT2rUL7TScHMrFDRbkkFiIi7gbs7lX2s4HMA70+HAbWtYhpj960b6MWamQ1qJfdEc5t9o2czuXm936tgZlagZJNCjJ/DaO1l2xY/2Wxm1qZkk8KIyckdSJvXPJlxJGZmg0fJJoVxM44FYOd6P8BmZtamZJPClNlH0xqiecvyrEMxMxs0SjYpVFaNZLMmUta4KutQzMwGjZJNCgBbK6YxerefVTAza1PSSWH3qFnUNHfuecPMrHSVdFJoGTeHCTS6C20zs1RJJ4WqmqS31E2rn8o4EjOzwaGkk8KY6ccAsGPd0owjMTMbHEo6KUyuPQ6A/Q2+LdXMDEo8KYwaM56tjCW/ze9VMDODEk8KAA3l06jevSbrMMzMBoWSTwo7R86i5kB91mGYmQ0KJZ8Umo84ikk8y64d27IOxcwscyWfFKqmJhebNyx/LONIzMyyV/JJ4YhZJwDQuHZJxpGYmWWv5JPCtCOPozlyNG3yswpmZiWfFMorqtiQm0Lldj+rYGZW8kkBYOuIWo7YuyrrMMzMMuekAOwbO5dpLetpbjqQdShmZpkqalKQdL6kpZKWSbqmi/rLJTVIWpwOf1/MeLqTn3Q0FWpm4+qns1i8mdmgUbSkICkPXA9cAMwHLpU0v4umP4iIU9LhW8WK52DGzEzC2rL68SwWb2Y2aBTzSOF0YFlErIiIA8CtwEVFXN4hmzrnJAD2bnAX2mZW2oqZFKYDhe+6rE/LOvs7SY9K+pGkmV3NSNIVkhZJWtTQ0NDvgY6ZMCnpGG/rM/0+bzOzoaSYSUFdlEWn8Z8BtRFxEvAb4NtdzSgiboiIuoioq6mp6ecwE5sqZjF6l3tLNbPSVsykUA8U/vKfAXR4IXJEbI2I/enoN4HTihjPQe0aNYcpTWuI1tasQjAzy1wxk8JCYJ6kOZIqgEuAOwsbSJpaMLoAeLKI8RxU1BzLeHaydZN7TDWz0lW0pBARzcBVwD0kX/a3RcQSSddJWpA2u1rSEkmPAFcDlxcrnp6MmnUyAOuffiirEMzMMldWzJlHxN3A3Z3KPlbw+Vrg2mLG0FvTjn4B/Br2rH0UeF3W4ZiZZcJPNKfG10xjK+PINWR2BsvMLHNOCgU2VM5h/C7flmpmpctJocCucccws3k1Lc3NWYdiZpYJJ4UC+SnHU6Um1q30C3fMrDQ5KRQYX3sKAFuW/zXjSMzMsuGkUGDGMafSGmL/Oh8pmFlpclIoUDVyNOtzU6h81h3jmVlpclLopGHkXCbuWZZ1GGZmmXBS6GT/EccyvXUDe3fvzDoUM7MB56TQSeWMk8krWPPUoqxDMTMbcE4KnUw97kwAti1fmHEkZmYDz0mhk8kzjmI7o8hteCTrUMzMBpyTQifK5aivnMcRO9wHkpmVHieFLuw64gRmNa9i//69WYdiZjagnBS6UDHzVCrUwtqnHs46FDOzAdVjUpA0V1Jl+vkcSVdLGlf80LIz+ZgzANi6zBebzay09OZI4XagRdJRwH8Dc4BbihpVxqbNOY6djCDWL846FDOzAdWbpNCavlrzdcCXIuJ9wNQephnSlMuztuIoxjX6YrOZlZbeJIUmSZcCbwPuSsvKixfS4LBj/PHMblpBU9OBrEMxMxswvUkKbwfOAj4ZESslzQG+W9ywslc+4xRG6ABrl7obbTMrHT0mhYh4IiKujojvSxoPjI6IzwxAbJmafOyLAGhY+kDGkZiZDZze3H30O0ljJB0BPALcJOkLxQ8tW9PnnkAj1VDvO5DMrHT05vTR2IjYAbweuCkiTgPOK25Y2VMuz+qq+dQ0Ppp1KGZmA6Y3SaFM0lTgYp670Nwrks6XtFTSMknXHKTdGySFpLq+zL/Y9kw6ldqWNexsfDbrUMzMBkRvksJ1wD3A8ohYKOlI4JmeJpKUB64HLgDmA5dKmt9Fu9HA1cCDfQl8IFQfeSY5Basf+1PWoZiZDYjeXGj+YUScFBH/kI6viIi/68W8TweWpe0PALcCF3XR7t+BzwH7+hD3gJh18ssA2Lns/owjMTMbGL250DxD0k8kbZa0SdLtkmb0Yt7TgbUF4/VpWeG8TwVmRsRBT0tJukLSIkmLGhoaerHo/jF2fA1rctOp2uTbUs2sNPTm9NFNwJ3ANJIv9Z+lZT1RF2XRXinlgC8CH+hpRhFxQ0TURURdTU1NLxbdfzaPOZFZe58gWlsHdLlmZlnoTVKoiYibIqI5HW4GevPNXA/MLBifAawvGB8NnAD8TtIq4EzgzsF2sbl1+guZQCPrVy3NOhQzs6LrTVLYIukySfl0uAzY2ovpFgLzJM2RVAFcQnLEAUBENEbExIiojYha4AFgQUQMqpcjTzz2xQCsX/L7jCMxMyu+3iSFd5DcjroR2AC8gaTri4NKO9G7iuTOpSeB2yJiiaTrJC049JAH1uzjXsguRtCy0hebzWz4K+upQUSsATp8iUt6L/ClXkx7N3B3p7KPddP2nJ7ml4V8WRkrRpzIlO0PZR2KmVnRHeqb197fr1EMcnumnkFt61q2bl6XdShmZkV1qEmhqzuLhq3xx50DwFdv+k62gZiZFdmhJoXoucnwMeekl7AvynnV6BVZh2JmVlTdXlOQtJOuv/wFjChaRINQRWUVS6rmM3HroLoxysys33V7pBARoyNiTBfD6Ijo8QL1cLNz0ukc2byCxu29uRvXzGxoOtTTRyVn9DFnk1ew8uF7sw7FzKxonBR6ae4LzuFA5NnztB9iM7Phy0mhl6pGjmZ55XxqGvwQm5kNX04KfbBj2ouZ27yCLZvW99zYzGwI6k3X2Tsl7eg0rE270z5yIIIcLI448VXkFKxcdHfPjc3MhqDeHCl8AfgQSbfZM4APAt8keWnOjcULbfA58uSXsoORtD5zX9ahmJkVRW+SwvkR8Y2I2BkROyLiBuDCiPgBML7I8Q0q+bJyVlSfyqztD/r9CmY2LPUmKbRKulhSLh0uLqgrqSebAZpmn81UGlizfEnWoZiZ9bveJIW3AG8FNqfDW4HLJI0g6Rq7pEx7wQUArH/4FxlHYmbW/3rTdfYK4DXdVP+pf8MZ/KbPPZGNqqFy1X3Ah7MOx8ysX/Xm7qMZ6Z1GmyVtknS7pBkDEdygJLEmN4Nj9jzE3j27s47GzKxf9eb00U0kr9GcRnIH0s/SspJ1Z+UCqrWfpQ/6FJKZDS+9SQo1EXFTRDSnw81ATZHjGtT+5Z+uYG9UsPdxP69gZsNLb5LCFkmXScqnw2VASXcVWjliFEur65i99Y++NdXMhpXeJIV3ABcDG4ENwBuAtxczqKGgee4rmcZmlj/hdyyY2fDRY1KIiDURsSAiaiJiUkS8Fnj9AMQ2qM0563UAbF7004wjMTPrP4faId77+zWKIWjCtFqWlR3FuPrfZh2KmVm/OdSkoF41ks6XtFTSMknXdFF/paTHJC2W9CdJ8w8xnkw8O+3lHNP0JA0b67MOxcysXxxqUuixewtJeeB64AJgPnBpF1/6t0TEiRFxCvA5ks73hozJZ7yRvIJv3/i1rEMxM+sX3T7RLGknXX/5CxjRi3mfDixLn4hG0q3ARcATbQ0iYkdB++puljdozZ5/OvW5abySB7IOxcysX3SbFCJi9GHOezqwtmC8HjijcyNJ/5vkGkUFcO5hLnNgSayb9ipOW/ttGjato2by9KwjMjM7LMV881pX1x2edyQQEddHxFzgn4GPdjkj6QpJiyQtamho6OcwD8+Usy6hTK0s+/2tWYdiZnbYipkU6oGZBeMzgIO9x/JW4LVdVUTEDRFRFxF1NTWD62Hq2fPPoD43lerld2UdipnZYStmUlgIzJM0R1IFcAlJH0rtJM0rGH018EwR4ykOifVTX8n8fYtp2LQu62jMzA5L0ZJCRDSTvG/hHuBJ4LaIWCLpOkkL0mZXSVoiaTHJdYW3FSueYprsU0hmNkwoYkjd8ENdXV0sWjTIupaIYO2/H8+O/HiO/8ifs47GzOx5JD0UEXU9tSvm6aPSIbGx9iKOb3qc1cuf6Lm9mdkg5aTQT+ac+05aQ9TfWvI9gJjZEOak0E8mzjiKJ6tOZnbzSlpb3J22mQ1NTgr96MDxb2JGbGTJg7/KOhQzs0PipNCPjnvFW9hDJbv/8p2sQzEzOyROCv2oqnosv9VZHL/tXnbs2JZ1OGZmfeak0M+OffVVjNZeHv/ljVmHYmbWZ04K/eyo085jVb6Wmqe+6/c3m9mQ46TQ3yTuKDufea0rWLLwvqyjMTPrEyeFInj3Vdeymyp2/fkbWYdiZtYnTgpFMGL0OJ6ceAGnNP6Whs0H6xjWzGxwcVIokimv+Eeq1MRTP78+61DMzHrNSaFIZhx3Ok9Uncoxq7/H3r17sw7HzKxXnBSKKPfi9zCJbTz88xuyDsXMrFecFIromBdfxKp8LVOXfIsW94dkZkOAk0IRKZej8dQrOTLW8PC9t2UdjplZj5wUiuyEV72DBk2g6i9fY6i90MjMSo+TQpHlyytZe+w7ObH5Mf76h59lHY6Z2UE5KQyAEy96L1s0noo/ftZdX5jZoOakMADKq6pZM/9KTmh+nId+f2fW4ZiZdctJYYCctOBqNsc4qn73cR8tmNmg5aQwQMoqR7LupKs4QSv4y29+mHU4ZmZdclIYQCctuJr1ualMvP8T7D9wIOtwzMyep6hJQdL5kpZKWibpmi7q3y/pCUmPSrpX0uxixpO1fHkl21/8EebGGh68/StZh2Nm9jxFSwqS8sD1wAXAfOBSSfM7NfsrUBcRJwE/Aj5XrHgGi/nnXsbSyhOYv/QrPLvt2azDMTProJhHCqcDyyJiRUQcAG4FLipsEBH3RcSedPQBYEYR4xkcJEa8+tNMpJHHvvaWrKMxM+ugmElhOrC2YLw+LevOO4FfdFUh6QpJiyQtamho6McQszHrpJexePyrOLP5Lyxd8teswzEza1fMpKAuyrrs50HSZUAd8Pmu6iPihoioi4i6mpqafgwxO3Mv+yIHVMHeO97jzvLMbNAoZlKoB2YWjM8AnvcaMknnAR8BFkTE/iLGM6iMnjCdlSd/kFOaHuHPd/xX1uGYmQHFTQoLgXmS5kiqAC4BOjzOK+lU4BskCWFzEWMZlE5c8B6WM5PjH/00Gzb6tZ1mlr2iJYWIaAauAu4BngRui4glkq6TtCBt9nlgFPBDSYsllVQfEMqXMfKSGxnDblb9v39wL6pmljkNtS+iurq6WLRoUdZh9KvF3/sIpzzzNf5w0md52euvzDocMxuGJD0UEXU9tfMTzYPAyZf8G8srjuWkR/6dtatXZB2OmZUwJ4VBQPlyxlz631TqAJu++y72NzVlHZKZlSgnhUGiZs4JrDz1GuqaFvGHGz+adThmVqKcFAaR+Qvez69yL+Hc9d/g/nvvyDocMytBTgqDicTLP3gLG8qmM+8PV7Ny5fKsIzKzEuOkMMiUjxxLxZu/S7X2sfs7l7KtcUfWIZlZCXFSGIQmzT2V9S//Eie0LuX+L76J/U3NWYdkZiXCSWGQmnv2m1ky//1cyP/w+2+83w+2mdmAcFIYxI5/48d4fPJFvHLLt/nFtz/rxGBmReekMJhJHP+ub/H06DM5f+Vn+NUPrs86IjMb5pwUBjmVVXDUVT9mRfVJnPvkx/jNHTdnHZKZDWNOCkNArrKa2qvuZKlqeelfP8Avb78x65DMbJhyUhgiykaO45gP/poNI47ivEc/wM9v+aqvMZhZv3NSGELKR01g5nt+zSM6jguW/gt3/vcnafZb28ysHzkpDDH5EWM49dpfszJfy0X1n+dXX343O/eWzAvrzKzInBSGoFxlNXM/spBnZl3MhTtu49H/XMC6zVuyDsvMhgEnhaEqX868t9/Airp/4czmB9n1f8/l/oUPZh2VmQ1xTgpDmcSRf/tBtrzmO0yJLZxw10X85LtfpcnXGczsEDkpDAOTT3sNlf90P9uqj+R1yz7K7dddzPJ1G7MOy8yGICeFYaJq4mxmfeD3LJ/3Ti7mN5Tf8BLuvONWWlp926qZ9Z6TwnCSL2fuW77Ajkt+SmVFBQsWv5sffvyNLF62OuvIzGyIcFIYhsYddzaTPrSQ5XPfxsX8hunfeQm33fAptuzcm3VoZjbIFTUpSDpf0lJJyyRd00X9yyQ9LKlZ0huKGUupUUU1c9/6Ffa9/TfsHz2bi9d/lvWffxG33f4Ddu5ryjo8MxukipYUJOWB64ELgPnApZLmd2q2BrgcuKVYcZS6kbV1zPjAH9l03leZVd7IxY9dweJPncvtd97BngN+eY+ZdVTMI4XTgWURsSIiDgC3AhcVNoiIVRHxKOB7KItJYvJL/hfj/vlxNpz+EU4pW83fPfw2HvjEedzyw1vZ3OjTSmaWKGZSmA6sLRivT8ssKxUjmXrhhxn9z0tYd9qHOL1sGW9e8m7W/+eL+M43v8BfV212J3tmJa6YSUFdlB3SN46kKyQtkrSooaHhMMMyKkcz/TUfZdQ1S9l69qeZMaKJt677OFNuOoPvf+YKbrvndzy7+0DWUZpZBsqKOO96YGbB+Axg/aHMKCJuAG4AqKur80/Z/lJRzYSX/yOcfSV7lvyc1j98gzc1/JD8/bfxwJ+PY0nNhUw47fWcc8rRjBtZkXW0ZjYAVKzTBZLKgKeBVwDrgIXAmyNiSRdtbwbuiogf9TTfurq6WLSys/YWAAAMeUlEQVRoUT9Ha+12rGfzn26m7JHvccT+epoiz59bT2BZzSsYc8pFnHn80cyaMDLrKM2sjyQ9FBF1PbYr5jlkSRcCXwLywI0R8UlJ1wGLIuJOSS8EfgKMB/YBGyPi+IPN00lhgEQQ6x6m4S+3Ub70TsbvX09LiMVxFI9W1dEy5+XMPvGlnDZnIkdU+yjCbLAbFEmhGJwUMhBBbHiE7Q//hOanf8OEHUvIEWyPau5vnc+qkSfSPP0Mphx7OqfNmUTthGpyua4uKZlZVpwUrHh2b+XAM/fS+Ng9VKy7n7H71gGwNyp4JObyuI5hx7jjyE87mSlzjmP+tPHMmzyKqvJ8xoGblS4nBRs4OzbQuuYBGpf+kdY1DzCu8SnytACwK6p4MmaxpLWWLVW1NI2fS/mkeUyYWkttzWhqJ1QzdWyVE4ZZkTkpWHaa98PmJ2nd8Ci7Vj1M8/pHqd7+FJUtu9ub7IlKVsUUVsRUVsVkdlRM5sCo6TB2BpUTZlEzYSI1oyuZOKqSCaMqmFBdyRHVFeR9WsrskDgp2OASATs3wNZlxJZn2LdxKfs3LqVs23JG7llPLj2yaLM9qlkfE1kXE2iIsWxhLFtjLPsqJ9A8YiJRPYn86MlUjRrPmJHljKkqZ8yItr9l7eOjKssYWZFnRHne1zmspPU2KRTzOQWz50gwZhqMmYbmvIwRwIi2utYW2LkRGuuhcS3RWM+IrauZ/ewaZu+op2zPasr3byNHa9Ihyu502AwHKKMxqpOB5O8Wqlmeju9o/zuSA/mRRHk1reUjoaKaqBhFrqKa8ooRjKwqp7IsR0VZjop8jsqyHOX5dDwte97ngrLyfDKey0FZLkc+B/lcjrxEPq/kb67jUJYTubQ8J5CctCx7TgqWvVwexk5PBs5AQGU6tGttgT1bYddm2N2QDLs2U7F7MxP3bmf8nm207N5G7N2O9q0ht7+RsqadqPND9K3A/nRItZBjL1XspZI9VLGbKvZEBXtaK9hPeftwIMrZQznb2sri+fVtn5soo5k8TZGnmTKaKKOJfFJGnuboWNZMntZcOcrlyStNGLlOfyXK0gSjNInkBOK5cUH6GXLpOO3tOk6D6DB9Lp0vHdo995n2dh2n4XnzTmPp0K5TnO3tnitPlvDc8jt7ro06jHecruP0HebSzfSFbXqzDDrNu2ObXsy707r1ZfqXHFXD/GljKCYnBRsacnkYNSkZOhHJP+Tn/WNubYX9jbAvHQ7s7mLYRf7AbkY17WHUgV0d6qJ5PzTvI5p2tX+meT9q2Yea9z8/4fSDVnK0qowWldGqHEGO1tYcrZGjlTytzTlalUvaUfg532E8EC1t5QXtWzj455YuyjvUhWhVnpZQe10LSXxJO9ESyZZpJUdriJZ0S7WEaEXt5a2RtNmxv4XWJEWk9UrXgWT907JIh2RbPLeMKKhP/uY6jHeYtm35iIiOZdFpmo4x5bqcTxKj6LpXn/5XO2ENv/vQy4u6DCcFG75yORgxPhkOgTr97SACWpvbE0XHv/ugpSkZWpugpRlaDjz3uTWtazmQzKOgXa7lALnWJspamiFakiOkDn9buyhv7aJdSxJjh7IDfZxHV+1autoah2cYPPsYCJRLD31yzw3PK+9cn0sPmZK20V6ngumfm6b1pR8q+ro4KZgdCgny5clQOTrraAZWd8mitRWIJMFEa5qUWjsORKfy6KK+sO5g9V206bK+4PPz6ruKgR7qo1N5oENdh9aWdDxZprqc/rnpciPHFn33OimYWd/k0l+4+fKsI7Ei8DuazcysnZOCmZm1c1IwM7N2TgpmZtbOScHMzNo5KZiZWTsnBTMza+ekYGZm7YZc19mSGoDVhzj5RGBLP4YzFHidS4PXuTQczjrPjoianhoNuaRwOCQt6k1/4sOJ17k0eJ1Lw0Css08fmZlZOycFMzNrV2pJ4YasA8iA17k0eJ1LQ9HXuaSuKZiZ2cGV2pGCmZkdhJOCmZm1K5mkIOl8SUslLZN0Tdbx9BdJMyXdJ+lJSUskvSctP0LSryU9k/4dn5ZL0lfS7fCopBdkuwaHRlJe0l8l3ZWOz5H0YLq+P5BUkZZXpuPL0vraLOM+VJLGSfqRpKfSfX1WCezj96X/ph+X9H1JVcNxP0u6UdJmSY8XlPV530p6W9r+GUlvO9R4SiIpSMoD1wMXAPOBSyXNzzaqftMMfCAijgPOBP53um7XAPdGxDzg3nQckm0wLx2uAL4+8CH3i/cATxaMfxb4Yrq+24B3puXvBLZFxFHAF9N2Q9GXgV9GxLHAySTrPmz3saTpwNVAXUScAOSBSxie+/lm4PxOZX3at5KOAP4VOAM4HfjXtkTSZxEx7AfgLOCegvFrgWuzjqtI6/pT4G+ApcDUtGwqsDT9/A3g0oL27e2GygDMSP+jnAvcRfLq8y1AWef9DdwDnJV+LkvbKet16OP6jgFWdo57mO/j6cBa4Ih0v90FvGq47megFnj8UPctcCnwjYLyDu36MpTEkQLP/QNrU5+WDSvpIfOpwIPA5IjYAJD+nZQ2Gw7b4kvAh4H0LetMALZHRHM6XrhO7eub1jem7YeSI4EG4Kb0lNm3JFUzjPdxRKwD/gNYA2wg2W8PMbz3c6G+7tt+2+elkhTURdmwuhdX0ijgduC9EbHjYE27KBsy20LS3wKbI+KhwuIumkYv6oaKMuAFwNcj4lRgN8+dTujKkF/n9NTHRcAcYBpQTXLqpLPhtJ97o7v17Lf1L5WkUA/MLBifAazPKJZ+J6mcJCF8LyJ+nBZvkjQ1rZ8KbE7Lh/q2eDGwQNIq4FaSU0hfAsZJKkvbFK5T+/qm9WOBZwcy4H5QD9RHxIPp+I9IksRw3ccA5wErI6IhIpqAHwMvYnjv50J93bf9ts9LJSksBOaldy5UkFywujPjmPqFJAH/DTwZEV8oqLoTaLsD4W0k1xrayv9XehfDmUBj22HqUBAR10bEjIioJdmPv42ItwD3AW9Im3Ve37bt8Ia0/ZD6BRkRG4G1ko5Ji14BPMEw3cepNcCZkkam/8bb1nnY7udO+rpv7wFeKWl8epT1yrSs77K+wDKAF3IuBJ4GlgMfyTqeflyvl5AcJj4KLE6HC0nOp94LPJP+PSJtL5I7sZYDj5Hc3ZH5ehziup8D3JV+PhL4C7AM+CFQmZZXpePL0vojs477ENf1FGBRup/vAMYP930MfBx4Cngc+A5QORz3M/B9kusmTSS/+N95KPsWeEe6/suAtx9qPO7mwszM2pXK6SMzM+sFJwUzM2vnpGBmZu2cFMzMrJ2TgpmZtXNSMDOzdk4KZr0g6RRJFxaML1A/dcEu6b2SRvbHvMwOl59TMOsFSZeTPCh0VRHmvSqd95Y+TJOPiJb+jsXMRwo2rEiqTV9C8830BS2/kjSim7ZzJf1S0kOS/ijp2LT8jemLXR6R9Ie0a5TrgDdJWizpTZIul/S1tP3Nkr6u5GVHKySdnb445UlJNxcs7+uSFqVxfTwtu5qkw7f7JN2Xll0q6bE0hs8WTL9L0nWSHgTOkvQZSU+kL1v5j+JsUSs5WT/i7cFDfw4k/dI3A6ek47cBl3XT9l5gXvr5DJL+ciDpPmB6+nlc+vdy4GsF07aPk7wk5VaSLgguAnYAJ5L86HqoIJa2rgrywO+Ak9LxVcDE9PM0kn5/akh6R/0t8Nq0LoCL2+ZF0pe+CuP04OFwBx8p2HC0MiIWp58fIkkUHaRdjb8I+KGkxSQvJZmaVv8ZuFnSu0i+wHvjZxERJAllU0Q8FhGtwJKC5V8s6WHgr8DxJG8B7OyFwO8i6R20Gfge8LK0roWkN1xIEs8+4FuSXg/s6WWcZgdV1nMTsyFnf8HnFqCr00c5khe2nNK5IiKulHQG8GpgsaTntTnIMls7Lb8VKJM0B/gg8MKI2JaeVqrqYj5d9YvfZl+k1xEiolnS6SS9h14CXEXSjbjZYfGRgpWkSF5EtFLSG6H9hegnp5/nRsSDEfExktc6zgR2AqMPY5FjSF6O0yhpMh1fGFM47weBsyVNTN8tfinw+84zS490xkbE3cB7SXpRNTtsPlKwUvYW4OuSPgqUk1wXeAT4vKR5JL/a703L1gDXpKeaPt3XBUXEI5L+SnI6aQXJKao2NwC/kLQhIl4u6VqS9wYIuDsifvr8OTIa+KmkqrTd+/oak1lXfEuqmZm18+kjMzNr59NHNuxJup7k3c6FvhwRN2URj9lg5tNHZmbWzqePzMysnZOCmZm1c1IwM7N2TgpmZtbu/wNgghevlYs7rgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "xgb7_2 = XGBClassifier(\n",
    "        learning_rate =0.01,\n",
    "        n_estimators=1000,  #数值大没关系，cv会自动返回合适的n_estimators\n",
    "        max_depth=5,\n",
    "        min_child_weight=2,\n",
    "        gamma=0,\n",
    "        subsample=0.8,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel=0.7,\n",
    "        objective= 'binary:logistic',\n",
    "        reg_alpha=0.1,\n",
    "        reg_lambda=0.5,\n",
    "        seed=3)\n",
    "\n",
    "modelfit(xgb7_2, X_train, y_train, cv_folds = kfold)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1000颗树还未收敛"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     test-logloss-mean  test-logloss-std  train-logloss-mean  \\\n",
      "0             0.647149          0.000030            0.647139   \n",
      "1             0.605536          0.000028            0.605520   \n",
      "2             0.567727          0.000043            0.567698   \n",
      "3             0.533234          0.000049            0.533187   \n",
      "4             0.501636          0.000055            0.501583   \n",
      "5             0.472631          0.000049            0.472565   \n",
      "6             0.445923          0.000054            0.445847   \n",
      "7             0.421292          0.000066            0.421192   \n",
      "8             0.398501          0.000065            0.398391   \n",
      "9             0.377398          0.000065            0.377274   \n",
      "10            0.357804          0.000070            0.357664   \n",
      "11            0.339603          0.000084            0.339443   \n",
      "12            0.322659          0.000083            0.322488   \n",
      "13            0.306869          0.000097            0.306680   \n",
      "14            0.292138          0.000095            0.291939   \n",
      "15            0.278376          0.000100            0.278166   \n",
      "16            0.265515          0.000091            0.265295   \n",
      "17            0.253498          0.000099            0.253254   \n",
      "18            0.242232          0.000102            0.241973   \n",
      "19            0.231693          0.000113            0.231424   \n",
      "20            0.221818          0.000117            0.221532   \n",
      "21            0.212551          0.000126            0.212244   \n",
      "22            0.203866          0.000129            0.203533   \n",
      "23            0.195692          0.000138            0.195345   \n",
      "24            0.188031          0.000141            0.187667   \n",
      "25            0.180828          0.000158            0.180440   \n",
      "26            0.174047          0.000151            0.173640   \n",
      "27            0.167680          0.000165            0.167251   \n",
      "28            0.161699          0.000155            0.161250   \n",
      "29            0.156067          0.000162            0.155590   \n",
      "..                 ...               ...                 ...   \n",
      "215           0.063399          0.001204            0.057762   \n",
      "216           0.063394          0.001209            0.057736   \n",
      "217           0.063390          0.001213            0.057713   \n",
      "218           0.063382          0.001223            0.057673   \n",
      "219           0.063371          0.001207            0.057640   \n",
      "220           0.063370          0.001202            0.057610   \n",
      "221           0.063372          0.001202            0.057579   \n",
      "222           0.063364          0.001205            0.057553   \n",
      "223           0.063362          0.001204            0.057517   \n",
      "224           0.063355          0.001203            0.057487   \n",
      "225           0.063359          0.001206            0.057444   \n",
      "226           0.063355          0.001212            0.057400   \n",
      "227           0.063357          0.001209            0.057366   \n",
      "228           0.063360          0.001209            0.057332   \n",
      "229           0.063358          0.001211            0.057311   \n",
      "230           0.063350          0.001216            0.057284   \n",
      "231           0.063351          0.001214            0.057260   \n",
      "232           0.063347          0.001218            0.057238   \n",
      "233           0.063344          0.001213            0.057221   \n",
      "234           0.063347          0.001218            0.057199   \n",
      "235           0.063353          0.001223            0.057166   \n",
      "236           0.063356          0.001220            0.057116   \n",
      "237           0.063362          0.001221            0.057088   \n",
      "238           0.063366          0.001218            0.057066   \n",
      "239           0.063365          0.001222            0.057042   \n",
      "240           0.063363          0.001218            0.057004   \n",
      "241           0.063349          0.001203            0.056952   \n",
      "242           0.063346          0.001205            0.056927   \n",
      "243           0.063342          0.001206            0.056893   \n",
      "244           0.063337          0.001200            0.056856   \n",
      "\n",
      "     train-logloss-std  \n",
      "0             0.000021  \n",
      "1             0.000016  \n",
      "2             0.000017  \n",
      "3             0.000014  \n",
      "4             0.000022  \n",
      "5             0.000025  \n",
      "6             0.000044  \n",
      "7             0.000043  \n",
      "8             0.000035  \n",
      "9             0.000042  \n",
      "10            0.000050  \n",
      "11            0.000055  \n",
      "12            0.000062  \n",
      "13            0.000056  \n",
      "14            0.000065  \n",
      "15            0.000062  \n",
      "16            0.000076  \n",
      "17            0.000080  \n",
      "18            0.000089  \n",
      "19            0.000078  \n",
      "20            0.000077  \n",
      "21            0.000092  \n",
      "22            0.000096  \n",
      "23            0.000095  \n",
      "24            0.000103  \n",
      "25            0.000096  \n",
      "26            0.000109  \n",
      "27            0.000104  \n",
      "28            0.000121  \n",
      "29            0.000123  \n",
      "..                 ...  \n",
      "215           0.000311  \n",
      "216           0.000297  \n",
      "217           0.000290  \n",
      "218           0.000296  \n",
      "219           0.000296  \n",
      "220           0.000296  \n",
      "221           0.000291  \n",
      "222           0.000293  \n",
      "223           0.000287  \n",
      "224           0.000284  \n",
      "225           0.000288  \n",
      "226           0.000288  \n",
      "227           0.000288  \n",
      "228           0.000288  \n",
      "229           0.000288  \n",
      "230           0.000289  \n",
      "231           0.000290  \n",
      "232           0.000293  \n",
      "233           0.000297  \n",
      "234           0.000294  \n",
      "235           0.000304  \n",
      "236           0.000298  \n",
      "237           0.000315  \n",
      "238           0.000313  \n",
      "239           0.000320  \n",
      "240           0.000328  \n",
      "241           0.000324  \n",
      "242           0.000333  \n",
      "243           0.000331  \n",
      "244           0.000323  \n",
      "\n",
      "[245 rows x 4 columns]\n",
      "logloss of train :\n",
      "0.057861313750023236\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEXCAYAAABCjVgAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XmcXnV99//X+7pmkskkk3UmZJlsZAECIksE3KliBaxstSgVb1Ert97lR6213mj92f7obavYWlvlp8UqWAUpatWAqLigKCowYFiSGAghJJN1sk62SWb53H+cM8OVYZYrk7nmzMz1fj4e53Gd8z3fc87ne87M9bnOrojAzMwMIJd1AGZmNnw4KZiZWRcnBTMz6+KkYGZmXZwUzMysi5OCmZl1cVIwKyDpo5L+I+s4zLLipDDCSJogab2kPy0oq5G0QdJbCsqWSbpH0m5JeyStkvQJSVPS8ddIape0P+3WSXp/iWM/X1JjKZdxLHqKJyL+ISL+rETLWy/pglLMuxSGanuNtPUy2jkpjDARsR+4FvhXSXVp8U1AQ0R8C0DSK4CfAw8CJ0fEZOBCoA14acHsfhMREyJiAvAW4CZJZw5NS+xYSKrIOgYrExHhbgR2wG3AN4DzgZ3AzIJxvwI+18/01wC/6lb2MPCnBcOXACuBPSRJ5pSCcaekZXvSOpcUjLsYWAXsAzYBHwLGA4eADmB/2s3qpV03A99Pp38IWFjE+jgZ+DGwC1gDXDmQeIC/A76eTjcfCOBdwEZgN/A+4GXAE2nbP1+wnIXAz9LtsQO4HZicjvtauqxD6bI+XMQ6Xg/873RZh4GKdHhT2pY1wOt7WBfnAVuBfEHZ5cATaf85QAPQDGwDPtPLOj0faOxl3CTgP4Em4HngY0AuHZcH/jldB88B16XrsaKXea0HLuhl3HuBtel2Xd75NwMI+BdgO7A3XUen9ba9s/5/HUld5gG4G+CGgynAlvQf710F5eOBduD8fqa/hoKkkH7R7QGWpMNLgAPAG4BK4MPpP+eYdHgt8NF0+HXpP+BJ6bRbgFcXxHlW2t/rl0xBHLelXwDnpF+CtwN39jPNeJIv7Xel05yVrpdTjzUeek4KXwSqgD8EWoDvAtOB2emX0mvT+ovS9TUWqAMeAD5bMO+jvvz6WscF9VcAc4BxwElpO2cVxNdjwgSeBd5QMPxN4Ia0/zfAO9L+CcB5vcyj1+1FkhC+B9SkcTwNvCcd9z6SL+X6dH3/hAEkhfTvake6PccCnwMeSMe9EXgUmEySIE4h/WHU2/Z2V1znw0cjVETsJvmFWQ38d8GoKSSHBbd2Fki6KT2vcEDSxwrqnpeW7yfZS/ga8Ew67q3A9yPixxHRCvwTyRfTK0h+iU4APhkRRyLiZ8A9wFXptK3AUkkTI2J3RDx2jM3774h4OCLaSJLCGf3U/yNgfUTcGhFt6fK+TXJIbDDi+fuIaImI+0i+xL8REdsjYhPwS+BMgIhYm66vwxHRBHwGeG0f8+1rHXf6t4jYGBGHSJL92LQtlRGxPiKe7WXe3yDdHpJqSH49f6NgfSySVBsR+yPit8eyMiTl09g/EhH7ImI9yZ7BO9IqVwL/GhGN6d/pJ49l/gXeDnwlIh6LiMPAR4CXS5qftqGGZA9REbE6IrYUtO94tndZc1IYoSRdTfIL7SfApwpG7SY5TDGzsyAiPhzJeYXvkPyS7vTbiJgcyTmFGcCpwD+k42aRHBbonEcHya/U2em4jWlZp+fTcQB/TPIl9LykX0h6+TE2b2tB/0GSBNSXecC5aYLbI2kPyRfKjEGKZ1tB/6EehicASJou6U5JmyQ1A18HavuYb1/ruNPGgvFrgQ+Q7M1sT5c1q5d53wFcIWkscAXwWER0Lus9JHspv5f0iKQ/6iPGntSS7CE+X1BWuP1nFcbdrf9YdF8/+0kOzc1Of4h8nuRQ4zZJt0iamFY93u1d1pwURiBJ00mOp74X+J/AlZJeAxARB0iOw19xLPOMiG0kv67fnBZtJvmy7VymSA5jbErHzZFU+PczNx1HRDwSEZeSHGL5LnBX52KOJaZjsBH4RZrgOrsJEfH+IY7nH9N5nh4RE4GrSQ5tdOq+vL7WcY/TRMQdEfGqdLrg6B8EhfVWkXyhXgT8KUmS6Bz3TERcRbI+PgV8S9L44pvJDpJf4/MKyrq2P8nhm/qCcXOOYd6Fuq+f8cA0Xvg7+7eIOJvkx8wS4K/T8t62txXBSWFk+jzw3Yi4P91l/jDwpfRXIenwuyXdkCYQJNUDC3qboaRpJCcjV6ZFdwFvkvR6SZXAX5Gc7Pw1SdI5AHxYUqWk80mSyZ2Sxkh6u6RJ6SGRZpLDHpD8wp4madIgrYdO9wBLJL0jjadS0ssknTLE8dSQnETeI2k26ZdUgW3AiQXDfa3jF5F0kqTXpdu5hWQvpb2nuqk7gOuB15CcU+icz9WS6tI9kz1pca/zkVRV2JHsid4FfCK9HHoe8EGSPaPOdv2FpNmSJpOcHO9PZbflVKTxv0vSGWmb/wF4KCLWp9v33HS9HUjXR3s/29uKkfVJDXfH1gGXkfyCmtyt/KfAJwqGzwXuJfmn3wM8BXwCmJaOv4bkn6XzypvtJMecpxfM43KSE4Z7gV+QnrhNx52alu1N61yelo8BfkhyGKsZeAR4VcF0XyE5BLCH3q8++j8Fw+fTz8nptN5JJFcsNaXz/xnJuYhjioeeTzRXFNRvpOAkPskX4ccK1smj6fpcQfIl31hQ91JgQ7qsDxWxjtdz9Inp00nO/ewjORl/T0/rsKD+XJIv8O93K/96ur33k/wIuKyX6c9P29+9W0Ry7urr6freCHycF64+qiDZk91JcvXRX5LsWaiX5azvYRn/Jx33PpKT5p3trU/LX09yxdF+XrjSa0J/29td/53SFWxmVhKSLgK+GBHz+q1smfPhIzMbVJLGSbpYUkV6GO1vSS5ysBHAewo2Ikh6NfCDnsZFcvWUDROSqkkOhZ1Mct7j+8BfRERzpoFZUZwUzMysiw8fmZlZlxH3kK3a2tqYP39+1mGYmY0ojz766I6IqOuv3ohLCvPnz6ehoSHrMMzMRhRJz/dfy4ePzMysgJOCmZl1cVIwM7MuTgpmZtbFScHMzLo4KZiZWRcnBTMz61I2SeHZJx/ioTv/kejo6L+ymVmZKpuk0PT4Dzj3959kX/PurEMxMxu2yiYpVEw8AYC9TY0ZR2JmNnyVTVIYOzl5h/u+nVsyjsTMbPgqm6QwfuosAFp2OymYmfWmbJLCxNokKbQ2b8s4EjOz4atsksKU2hl0hIh927MOxcxs2CqbpJCvqGSPatDBpqxDMTMbtsomKQDszU1hTMuOrMMwMxu2yiopHKicQvWRXVmHYWY2bJVVUmgZW0tNu29eMzPrTVklhbZxtUzu2JN1GGZmw1ZZJYUYP51qHebAPicGM7OelFVSyNdMB2BP06aMIzEzG55KmhQkXShpjaS1km7opc6VklZJWinpjlLG0/Woix2bS7kYM7MRq6JUM5aUB24G3gA0Ao9IWh4RqwrqLAY+ArwyInZLml6qeADGT50JwKHdTgpmZj0p5Z7COcDaiFgXEUeAO4FLu9V5L3BzROwGiIiS3m48+YR5ABxxUjAz61Epk8JsYGPBcGNaVmgJsETSg5J+K+nCnmYk6VpJDZIampoGfkfy1LpZtEWOjmY/FM/MrCelTArqoSy6DVcAi4HzgauA/5A0+UUTRdwSEcsiYlldXd2AA8rl8+zUFCoObB3wPMzMRrNSJoVGYE7BcD3Q/bhNI/C9iGiNiOeANSRJomT2VNRS1eKH4pmZ9aSUSeERYLGkBZLGAG8Dlner813gDwAk1ZIcTlpXwpg4OLaOmlY//8jMrCclSwoR0QZcB/wIWA3cFRErJd0o6ZK02o+AnZJWAfcDfx0RO0sVE8CRcScwtaOkizAzG7FKdkkqQETcC9zbrezjBf0BfDDthkRHzQwm7jjIwf17qZ4waagWa2Y2IpTVHc0AFZOSN7Dt3Lqxn5pmZuWn7JJC1dTk3Hfz9g0ZR2JmNvyUXVKYWFcPwKFdjRlHYmY2/JRdUpgyM7mruXWP72o2M+uu7JJCzcSpHIyxaJ/vajYz667skoJyOXblplJ5cFvWoZiZDTtllxQA9lbWUu27ms3MXqQsk8KhsdOZ2Oa7ms3MuivLpNA2/gSmdewiOjqyDsXMbFgpy6RAzUyq1Erz7oE/htvMbDQqy6RQOTm5q3nXNt/AZmZWqCyTQnVtclfzviY/6sLMrFBZJoWJdXMBaPFdzWZmRynLpDBtRrKn0O67ms3MjlKWSaGqegJ7mEBuv+9qNjMrVJZJAWB3bhpjDvkGNjOzQmWbFPaNqWPCYScFM7NCZZsUWqpnMq3dScHMrFDZJoWOiXOZSjMH9+/NOhQzs2GjbJNC5bTkstTtjc9mHImZ2fBRtklh/PQTAdi7xUnBzKxT2SaFqbMXAXCoaX22gZiZDSNlmxRqZ8zlSOSJ3X7+kZlZp7JNCrmKCppytVTu96MuzMw6lTQpSLpQ0hpJayXd0MP4ayQ1SVqRdn9Wyni62z1mJjWH/KgLM7NOFaWasaQ8cDPwBqAReETS8ohY1a3qf0XEdaWKoy+Hxs3ihD2/yWLRZmbDUin3FM4B1kbEuog4AtwJXFrC5R2ztolzqGM3LYcOZB2KmdmwUMqkMBsofGFBY1rW3R9LekLStyTNKWE8L1Ix1fcqmJkVKmVSUA9l0W34bmB+RJwO/AT4ao8zkq6V1CCpoalp8F6hWd11r8K6QZunmdlIVsqk0AgU/vKvB446qxsROyPicDr4JeDsnmYUEbdExLKIWFZXVzdoAU6dvRCAg03PDdo8zcxGslImhUeAxZIWSBoDvA1YXlhB0syCwUuA1SWM50XqZi2gLXJ07PK9CmZmUMKrjyKiTdJ1wI+APPCViFgp6UagISKWA9dLugRoA3YB15Qqnp5UVI5hq6ZRsc/vajYzgxImBYCIuBe4t1vZxwv6PwJ8pJQx9GfXmBmMP+Q3sJmZQRnf0dzpwLiZTG3dmnUYZmbDQtknhfaaeupiJ61HDvdf2cxslCv7pJCfMo+8gqZNvgLJzKzsk8K46QsA2LXpmYwjMTPLXtknhan1JwFwcKuTgplZ2SeFE+Ys4khU0L5jbdahmJllruyTQr6igi35mVQ1r886FDOzzJV9UgDYXTWHyS2+gc3MzEkBaJk4n5ntW+hob886FDOzTDkpALlpC6lSK9v8CG0zK3NOCsD4mckVSDs2DOnz+MzMhh0nBaBu/qkAHNy8JuNIzMyy5aQA1M2ax6EYQ+z04SMzK29OCoByebbmZ1G1b33WoZiZZcpJIbVn3Bym+rJUMytzTgqpw5MWMKNjK22tR7IOxcwsM04KqXztIsaonW0b/bgLMytfTgqpCbN9WaqZmZNCqm7eUgAObfFlqWZWvpwUUtOm13MgqmCXL0s1s/LlpJBSLsfWitmM2/d81qGYmWXGSaHAnvHzmd6yPuswzMwy46RQoHXqycykiX17d2UdiplZJpwUCoybnTwDadMzKzKOxMwsG/0mBUkLJY1N+8+XdL2kyaUPbejVnXgGAHuffzLjSMzMslHMnsK3gXZJi4AvAwuAO4qZuaQLJa2RtFbSDX3Ue4ukkLSsqKhLZMa8k5IH421blWUYZmaZKSYpdEREG3A58NmI+EtgZn8TScoDNwMXAUuBqyQt7aFeDXA98NCxBF4KuYoKNlXMoXrvM1mHYmaWiWKSQqukq4B3AvekZZVFTHcOsDYi1kXEEeBO4NIe6v09cBPQUsQ8S273hEXMOPxc1mGYmWWimKTwLuDlwCci4jlJC4CvFzHdbKDwsaONaVkXSWcCcyLiHvog6VpJDZIampqailj0wHXUnsR0drF31/aSLsfMbDjqNylExKqIuD4iviFpClATEZ8sYt7qaXZdI6Uc8C/AXxURwy0RsSwiltXV1RWx6IGrqn8JAJue/l1Jl2NmNhwVc/XRzyVNlDQVeBy4VdJniph3IzCnYLge2FwwXAOcBvxc0nrgPGB55iebFyZXIDVv8BVIZlZ+ijl8NCkimoErgFsj4mzggiKmewRYLGmBpDHA24DlnSMjYm9E1EbE/IiYD/wWuCQiGo65FYNoev2i5BlI230FkpmVn2KSQoWkmcCVvHCiuV/pFUvXAT8CVgN3RcRKSTdKumRA0Q4B5XJsqpzHhGa/V8HMyk9FEXVuJPlifzAiHpF0IlDUNZsRcS9wb7eyj/dS9/xi5jkU9tYs4sTdvyIikHo6NWJmNjoVc6L5mxFxekS8Px1eFxF/XPrQstNeezLT2MvO7Zv7r2xmNooUc6K5XtJ3JG2XtE3StyXVD0VwWZkwJ7kCaeszj2YciZnZ0CrmnMKtJCeIZ5HcZ3B3WjZqzT75HAD2PfdYxpGYmQ2tYpJCXUTcGhFtaXcbUNqbBTI2ZfpstjGNiu2+LNXMyksxSWGHpKsl5dPuamBnqQPL2pbqkzhh/+qswzAzG1LFJIV3k1yOuhXYAryF5NEXo1pL7Uuo79jsF+6YWVkp5uqjDRFxSUTURcT0iLiM5Ea2Ua16/lnkFGxYlfnDW83MhsxA37z2wUGNYhiafcorAGhe5yuQzKx8DDQpjPo7uqbNnMsOJpPf9kTWoZiZDZmBJoXov8rIt2ncSdT5ZLOZlZFeH3MhaR89f/kLGFeyiIaRQ7WncdqGhzmwv5nxEyZmHY6ZWcn1uqcQETURMbGHriYiinlm0og3bu7Z5BU8v+rhrEMxMxsSAz18VBZmLT0PgL3rHsk4EjOzoeGk0IfamQvYxURyWx/POhQzsyHhpNAH5XJsGncS05tXZh2KmdmQcFLox8ETzmZe+0b27mrKOhQzs5Ir5tHZ+yQ1d+s2po/TPnEogszShEWvJKdg/eMPZB2KmVnJFbOn8Bngr0kem10PfAj4EnAn8JXShTY8LHjpq2kPceDZB7MOxcys5IpJChdGxL9HxL6IaI6IW4CLI+K/gCklji9z1TVTeL5iATXb/bgLMxv9ikkKHZKulJRLuysLxpXFnc07pp7JiYd/T2vrkaxDMTMrqWKSwtuBdwDb0+4dwNWSxgHXlTC2YaNi3nmMVwvrVzVkHYqZWUkV8+jsdRHx5oioTbs3R8TaiDgUEb8aiiCzVn/6awHYsdonm81sdCvm6qP69Eqj7ZK2Sfq2pPqhCG64mD5nCU1MIb/Jdzab2ehWzOGjW4HlwCySK5DuTsvKh0TjhJdQv8+P0Taz0a2YpFAXEbdGRFva3QbUFTNzSRdKWiNpraQbehj/PklPSloh6VeSlh5j/EOmddbLmMV2tjY+l3UoZmYlU0xS2CHpakn5tLsa2NnfRJLywM3ARcBS4KoevvTviIiXRMQZwE0k90QMS9NOSc4rbFzx04wjMTMrnWKSwruBK4GtwBbgLcC7ipjuHGBteqL6CMnNbpcWVoiI5oLB8QzjS1znv+QV7GMc7c/+POtQzMxKpt/3IkTEBuCSwjJJHwA+28+ks4GNBcONwLndK0n6c5J3Po8BXtdfPFnJV1SyrvpM5ux5mIhAGvVvJDWzMjTQB+J9sIg6PX1rvmhPICJujoiFwP8GPtbjjKRrJTVIamhqyu7BdEfmvorZsY1Nz63JLAYzs1IaaFIo5mdyIzCnYLge2NxH/TuBy3oaERG3RMSyiFhWV1fUOe6SmHHGGwHY+NgPM4vBzKyUBpoUijn2/wiwWNICSWOAt5Fc2tpF0uKCwTcBzwwwniFRv+QsdjKZiud9E5uZjU69nlOQtI+ev/wFjOtvxhHRJuk64EdAHvhKRKyUdCPQEBHLgeskXQC0AruBdw6gDUNGuRzPTzyb+c2P0tHeQS7v11GY2ejSa1KIiJrjnXlE3Avc263s4wX9f3G8yxhqseC11D3+U9aubmDRaedkHY6Z2aDyT91jNPfsiwDY/vh9GUdiZjb4nBSOUd3cJWzWDMY1/jLrUMzMBp2TwgBsqn0FJx38HQcP7s86FDOzQeWkMADVp72Jah3m97/+ftahmJkNKieFAVh87sUcjLEcXnVv/5XNzEYQJ4UBGFNVzTMTlrFg16/oaO/IOhwzs0HjpDBAbYveyAx28MyTv806FDOzQeOkMEAnvuIKAJoe+17GkZiZDR4nhQGacsIcnqlYQu2m+7MOxcxs0DgpHIfd9a9nSdvTbNu8IetQzMwGhZPCcZhxzhXkFDz7y//KOhQzs0HhpHAc5p7yMjbm6qlZu7z/ymZmI4CTwvGQ2DrnIk498iSbG5/LOhozs+PmpHCcZr/q7eQUrPvFHVmHYmZ23JwUjtOsxWeyPj+fqc/dnXUoZmbHzUlhEDTN/yOWtq1mo9/dbGYjnJPCIJj76qsB2PDA7RlHYmZ2fJwUBsEJ809hLfVMX/dtIop5fbWZ2fDkpDBImk97J4vVyOrH/PIdMxu5nBQGyUkXvJuWqGTPr76cdShmZgPmpDBIxk+uZfWU8zlt133s29ecdThmZgPipDCIal7xHibqIE/++GtZh2JmNiBOCoNo4bI3sik3k5rVd2YdipnZgDgpDCLlcmxZ8BZe0voEa1etyDocM7Nj5qQwyJa88X0ciQq2/uTfsg7FzOyYlTQpSLpQ0hpJayXd0MP4D0paJekJST+VNK+U8QyFidPreWrqGzhz5z3saNqedThmZsekZElBUh64GbgIWApcJWlpt2q/A5ZFxOnAt4CbShXPUJr+hg8wXodZ+f3PZR2KmdkxKeWewjnA2ohYFxFHgDuBSwsrRMT9EXEwHfwtUF/CeIZM/dLz+H3VS1my/g5aDh/OOhwzs6KVMinMBjYWDDemZb15D/CDnkZIulZSg6SGpqamQQyxdOK8/8VMdvDYD7+adShmZkUrZVJQD2U9PhhI0tXAMuDTPY2PiFsiYllELKurqxvEEEvn5Nf8CZtyM5n6+L/T3t6RdThmZkUpZVJoBOYUDNcDm7tXknQB8DfAJRExao61KJdn15nXcXLHWh75se9bMLORoZRJ4RFgsaQFksYAbwOOepmxpDOBfydJCKPuUp1TL7yWLbkTmPzwP3tvwcxGhJIlhYhoA64DfgSsBu6KiJWSbpR0SVrt08AE4JuSVkha3svsRqRc5Rh2nHk9J3es5eH7vLdgZsOfRtrz/5ctWxYNDQ1Zh1G0jtYjbPvH09hLDYs++jAVFfmsQzKzMiTp0YhY1l8939FcYoV7C7+59z+zDsfMrE9OCkPgtIv/Jxvzc5n/2CfZf/Bg/xOYmWXESWEIKF9J6wV/zxy28vB/fTLrcMzMeuWkMEROfPllrB5/DsvWf4lNmzb2P4GZWQacFIZQ7RU3UU0Lz9710axDMTPrkZPCEKpbeCar6v+EV+25m4cf+GHW4ZiZvYiTwhA75e2fZmduGtPu/xDNBw5kHY6Z2VGcFIZYZfUk9l1wEwtjIw9/7eNZh2NmdhQnhQyc+Mo/ZuWUC3j1lttY0fBg1uGYmXVxUsjIgv/xeQ7mxjPhnvexa8/erMMxMwOcFDJTPWUmzRd+jkVs4EefvZaR9rgRMxudnBQyNO/cS3lq7tVcxQ/56ff8Mh4zy56TQsZOfcc/8/yYhbzsdx9lxYqR86A/MxudnBQypsoqprzrLsjlmPjdd9K4ZVvWIZlZGXNSGAYmzlzEgUu+zFw20/jlq9nfciTrkMysTDkpDBOzznwjzy37fzmv7WF+/bl3c7i1LeuQzKwMOSkMI4v/6IOsWfgu/vDA3dz3hb+ivcNXJJnZ0HJSGGZOuvpfeHrmJbx5120sv+VvnRjMbEg5KQw3Ekv+7FaenfpaLt/6ryz/4t/Q1t6RdVRmViacFIajfAUL/9e3eLb2dVy+/Wbu/v8/zOG29qyjMrMy4KQwXFWMYeH7v8mzFYu5fOeXuO8zf8bufYeyjsrMRjknheEsX8HCjz7EuoXv4M0H/5snP3sZ6zY3ZR2VmY1iTgrDXS7Pie/4PBvO/Vte2fYQh774eu7/9UNZR2Vmo5STwggx96IPsvuyrzE3v5Ozf3QZ3/jqzbS0+jyDmQ2ukiYFSRdKWiNpraQbehj/GkmPSWqT9JZSxjIa1J75Zsb++YPsHz+Pq577KD+56SqefLYx67DMbBQpWVKQlAduBi4ClgJXSVrardoG4BrgjlLFMdqMqZ3PrA/+go2nvJeLj9zHpK+ez+13fp0Dh30HtJkdv1LuKZwDrI2IdRFxBLgTuLSwQkSsj4gnAF+IfywqxjLnrf/EwavvobpqDG///Z/zwKeu4CcPP+H3MpjZcSllUpgNbCwYbkzLjpmkayU1SGpoavLVN50mLH4VtR9qYMtLr+P17Q9yzvf/kNs//f/wy5XrnRzMbEBKmRTUQ9mAvqki4paIWBYRy+rq6o4zrFFmTDUzL/8E+et+y74Z53H1wa9x8l2v5tbP3MCv12xycjCzY1LKpNAIzCkYrgc2l3B5ZS1ft5jZ7/8urdfcR9u0k3n3vi+y4I5Xceunruc7Dz7pK5XMrCilTAqPAIslLZA0BngbsLyEyzOgcv65zLz+xxz50++guiW8u+U/ufC+1/HtG6/klm8uZ9XmZu89mFmvVMovCEkXA58F8sBXIuITkm4EGiJiuaSXAd8BpgAtwNaIOLWveS5btiwaGvzaymLF1qdo+vFnmbLuu1RGK091zOeX1W9g3Flv5Q/OXsq8aeOzDtHMhoCkRyNiWb/1RtqvRieFATqwgwOP3klLw+1Ma15FW+T4TcdSVox/JflT3sS5Z7yE0+snU5n3/Yxmo5GTgvVu2yqaH76dWH03kw4+D8ATHQt4SKezu+4cJp30as5aPIfT6ycxtiKfcbBmNhicFKw4TU/T8tRyDj71fSbtfJw87bRGnifiRB5hKbsnnYZmn0X9vEWcVj+Zk2fUUFXpRGE20jgp2LE7cgA2PsShp39Oy9oHmLjrCfKRXLW0IybyVMcCVsZ8dlYvoHXqEqpOOIn6E2pZUDueBbXjmTmpigoffjIblpwU7Pi1HoJtK4nNv+PA+gY6Nv2OCXvXkuOFy1sbo5Z1HTN5NmaxMaZzYNwM2ibUk5syh5opM5g5eRx1NWOZOn4MU8ePYdqEMUypHuO9DbMh5qRgpdEYDyrHAAAKRklEQVR2GHatgx1PE01rOLzl97RtX8PYvc9S2X70S4AOxRg2xzQ2RS3bYgo7mcSOmMjOmMiBism0VdfSMa6WjuppVFdVU1NVwYSqCmqqKqkZW0FN2l89Nk9VRZ6xlTmqKvJUVeaoqsynXVKWy/V0r6SZdSo2KVQMRTA2ilSMhemnwPRTEFDVWR4BB3fB3o2wtxH2NlK1dyNzd21g9u6N5PY/TUXLTnIdrS/MqyXtdsN+xtPMePZ0jGMf42iOavZRzY4Yxz6qu4YPxlhaGMtBxnIoxnCIsRxiLG25Ktorq6FiHLnKsVTm8+RzoiKfoyInKvJKPnO5F/q7xuWozOlF9SvzOfI5peOS6fI5kRMIIYHUOQy5nBBJmQQ5JcM5JRVynXXTftJxhXVfNG3uxcvqsa4gyYudy+geZ+fy+65bSH3k2cJx6lZRvdRLxvW+jBct7qhxxU9XGM+Lx/U+z6OW1/uoAbe32PXZ13R5qeQ/gJwUbHBIMH5a0s06IykCKtMOSBLH4WY4sCPtmtJuBxMONDGhZQ+zWpqJlr10tDQTLU3ocDO5I/tQFPnMxDboaMtxRGNp1RjaVEmrKmmlglaS/iNUcIRKjkQFrVRwOO0/HBW0UMm+1hwtUUFb5Gklz2HytJGjnTxtaddOjrbIp2XJuFaOHm6LHG1UHDVtO7lkHnH0cOdnBznaydGBiOSrf/C3lY1Y86dV8/O//oOSLsNJwYaOBFWTkm7awt6rkdzt2CUiOQl+uDk5z3HkQPLZmn4eOQitL3S5Iwepaj1IVdthaD8M7a3JYa/2Iy98dvUfTD9bk7pthyGfjovsHw0SygE5Ipd+KkcoD1LyiV5UFqSfyiWJRXmicFzXfNIuTUIo322cCF6Y9oVl5dJ6+XT+hWXdp+1WTjIO5eggGUY5OqS0fWl8nfVIxxVMmyTLZH4d6pzH0dMl/XqhX53DuYJxuRfWVed0ytERemF+BdOD6OhcH53Lh67YOl60nKOXd9R2LXgMXF9H8Lsf3n/tkumD8FfVNycFG/4kGDsh6YZSR0eSGDrakq69FToKhju6D7dBe9vRwx3tab2C4fbWPuq0J98SkSxb0QEd6WcUjOto76qTfHak8XYvK+h/UVnnZxSUtXarF0fP46hpe4mlo3B57UfXGdgzMUeBJLEd1eXyaX/3cfluwwXj6z4K9aV9H5mTgllv0l/n5Cv7rWpF6kp4cXTyOqqLXsq71ekp4R3TPIpcBt2TX/fl9BFHR2/zLkz+PSXsntrQDtVTS76JnBTMbOgoOUyV8NfPcOQ7jczMrIuTgpmZdXFSMDOzLk4KZmbWxUnBzMy6OCmYmVkXJwUzM+vipGBmZl1G3KOzJTUBzw9w8lpgxyCGM5KUa9vd7vJTrm3vr93zIqKuv5mMuKRwPCQ1FPM88dGoXNvudpefcm37YLXbh4/MzKyLk4KZmXUpt6RwS9YBZKhc2+52l59ybfugtLuszimYmVnfym1PwczM+uCkYGZmXcomKUi6UNIaSWsl3ZB1PKUkab2kJyWtkNSQlk2V9GNJz6SfU7KOczBI+oqk7ZKeKijrsa1K/Fv6N/CEpLOyi/z49NLuv5O0Kd3uKyRdXDDuI2m710h6YzZRHz9JcyTdL2m1pJWS/iItH9XbvI92D/42j4hR35G8B/5Z4ERgDPA4sDTruErY3vVAbbeym4Ab0v4bgE9lHecgtfU1wFnAU/21FbgY+AEg4DzgoazjH+R2/x3woR7qLk3/5scCC9L/hXzWbRhgu2cCZ6X9NcDTaftG9Tbvo92Dvs3LZU/hHGBtRKyLiCPAncClGcc01C4Fvpr2fxW4LMNYBk1EPADs6lbcW1svBf4zEr8FJkuaOTSRDq5e2t2bS4E7I+JwRDwHrCX5nxhxImJLRDyW9u8DVgOzGeXbvI9292bA27xcksJsYGPBcCN9r9CRLoD7JD0q6dq07ISI2ALJHxgwPbPoSq+3tpbD38F16WGSrxQcIhyV7ZY0HzgTeIgy2ubd2g2DvM3LJSmoh7LRfC3uKyPiLOAi4M8lvSbrgIaJ0f538AVgIXAGsAX457R81LVb0gTg28AHIqK5r6o9lI3YtvfQ7kHf5uWSFBqBOQXD9cDmjGIpuYjYnH5uB75Dstu4rXO3Of3cnl2EJddbW0f130FEbIuI9ojoAL7EC4cLRlW7JVWSfDHeHhH/nRaP+m3eU7tLsc3LJSk8AiyWtEDSGOBtwPKMYyoJSeMl1XT2A38IPEXS3nem1d4JfC+bCIdEb21dDvyP9IqU84C9nYccRoNux8ovJ9nukLT7bZLGSloALAYeHur4BoMkAV8GVkfEZwpGjept3lu7S7LNsz6rPoRn7y8mOWP/LPA3WcdTwnaeSHLVwePAys62AtOAnwLPpJ9Ts451kNr7DZLd5laSX0fv6a2tJLvUN6d/A08Cy7KOf5Db/bW0XU+kXwozC+r/TdruNcBFWcd/HO1+FclhkCeAFWl38Wjf5n20e9C3uR9zYWZmXcrl8JGZmRXBScHMzLo4KZiZWRcnBTMz6+KkYGZmXZwUzMysi5OCWREkndHtscSXDNYj2CV9QFL1YMzL7Hj5PgWzIki6huTGp+tKMO/16bx3HMM0+YhoH+xYzLynYKOKpPnpi0i+lL6M5D5J43qpu1DSD9Onyf5S0slp+Z9IekrS45IeSB+NciPw1vRFJm+VdI2kz6f1b5P0hfQlKOskvTZ9YuVqSbcVLO8LkhrSuP6/tOx6YBZwv6T707KrlLwk6SlJnyqYfr+kGyU9BLxc0iclrUqfkPlPpVmjVnayvn3bnbvB7ID5QBtwRjp8F3B1L3V/CixO+88Ffpb2PwnMTvsnp5/XAJ8vmLZrGLiN5B0dInmOfTPwEpIfXY8WxNL56IU88HPg9HR4PelLkUgSxAagDqgAfgZclo4L4MrOeZE8vkCFcbpzd7yd9xRsNHouIlak/Y+SJIqjpI8gfgXwTUkrgH8nebsVwIPAbZLeS/IFXoy7IyJIEsq2iHgykidXrixY/pWSHgN+B5xK8nas7l4G/DwimiKiDbid5C1rAO0kT8mEJPG0AP8h6QrgYJFxmvWpIusAzErgcEF/O9DT4aMcsCcizug+IiLeJ+lc4E3ACkkvqtPHMju6Lb8DqEifVPkh4GURsTs9rFTVw3x6eg5+p5ZIzyNERJukc4DXkzz19zrgdUXEadYn7ylYWYrkBSXPSfoT6HrB+0vT/oUR8VBEfBzYQfJc+n0k78YdqInAAWCvpBNIXoDUqXDeDwGvlVQrKQ9cBfyi+8zSPZ1JEXEv8AGSl6yYHTfvKVg5ezvwBUkfAypJzgs8Dnxa0mKSX+0/Tcs2ADekh5r+8VgXFBGPS/odyeGkdSSHqDrdAvxA0paI+ANJHwHuT5d/b0T09O6LGuB7kqrSen95rDGZ9cSXpJqZWRcfPjIzsy4+fGSjnqSbgVd2K/7XiLg1i3jMhjMfPjIzsy4+fGRmZl2cFMzMrIuTgpmZdXFSMDOzLv8Xa6DHVRbUE5oAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "xgb7_3 = XGBClassifier(\n",
    "        learning_rate =0.05,\n",
    "        n_estimators=1000,  #数值大没关系，cv会自动返回合适的n_estimators\n",
    "        max_depth=5,\n",
    "        min_child_weight=2,\n",
    "        gamma=0,\n",
    "        subsample=0.8,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel=0.7,\n",
    "        objective= 'binary:logistic',\n",
    "        reg_alpha=0.1,\n",
    "        reg_lambda=0.5,\n",
    "        seed=3)\n",
    "\n",
    "modelfit(xgb7_3, X_train, y_train, cv_folds = kfold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    test-logloss-mean  test-logloss-std  train-logloss-mean  train-logloss-std\n",
      "0            0.523193          0.000109            0.523154           0.000072\n",
      "1            0.409306          0.000072            0.409230           0.000035\n",
      "2            0.328190          0.000118            0.328025           0.000050\n",
      "3            0.268299          0.000126            0.268059           0.000067\n",
      "4            0.223041          0.000139            0.222773           0.000081\n",
      "5            0.188414          0.000127            0.188053           0.000131\n",
      "6            0.161614          0.000112            0.161164           0.000163\n",
      "7            0.140792          0.000170            0.140240           0.000177\n",
      "8            0.124462          0.000193            0.123802           0.000151\n",
      "9            0.111673          0.000187            0.110915           0.000183\n",
      "10           0.101601          0.000209            0.100727           0.000202\n",
      "11           0.093671          0.000240            0.092662           0.000224\n",
      "12           0.087428          0.000298            0.086296           0.000252\n",
      "13           0.082545          0.000341            0.081277           0.000279\n",
      "14           0.078727          0.000363            0.077326           0.000284\n",
      "15           0.075725          0.000380            0.074169           0.000281\n",
      "16           0.073331          0.000441            0.071643           0.000293\n",
      "17           0.071459          0.000500            0.069614           0.000297\n",
      "18           0.070000          0.000527            0.067957           0.000320\n",
      "19           0.068888          0.000541            0.066691           0.000323\n",
      "20           0.067995          0.000558            0.065641           0.000304\n",
      "21           0.067296          0.000586            0.064769           0.000404\n",
      "22           0.066711          0.000641            0.064025           0.000395\n",
      "23           0.066218          0.000665            0.063408           0.000419\n",
      "24           0.065852          0.000670            0.062926           0.000425\n",
      "25           0.065594          0.000678            0.062521           0.000434\n",
      "26           0.065333          0.000715            0.062148           0.000422\n",
      "27           0.065101          0.000740            0.061850           0.000432\n",
      "28           0.064921          0.000758            0.061584           0.000417\n",
      "29           0.064784          0.000811            0.061367           0.000413\n",
      "30           0.064624          0.000822            0.061114           0.000408\n",
      "31           0.064527          0.000829            0.060929           0.000423\n",
      "32           0.064479          0.000875            0.060773           0.000437\n",
      "33           0.064419          0.000927            0.060600           0.000413\n",
      "34           0.064393          0.000951            0.060383           0.000455\n",
      "35           0.064302          0.000992            0.060159           0.000393\n",
      "36           0.064234          0.000991            0.059991           0.000394\n",
      "37           0.064175          0.000989            0.059839           0.000392\n",
      "38           0.064145          0.001015            0.059731           0.000400\n",
      "39           0.064151          0.001003            0.059586           0.000388\n",
      "40           0.064051          0.001098            0.059430           0.000237\n",
      "41           0.063990          0.001113            0.059321           0.000234\n",
      "42           0.064016          0.001121            0.059197           0.000209\n",
      "43           0.063948          0.001106            0.059044           0.000205\n",
      "44           0.063918          0.001123            0.058955           0.000177\n",
      "45           0.063897          0.001107            0.058857           0.000163\n",
      "46           0.063910          0.001063            0.058695           0.000228\n",
      "47           0.063894          0.001070            0.058567           0.000191\n",
      "48           0.063871          0.001087            0.058431           0.000195\n",
      "49           0.063880          0.001095            0.058341           0.000203\n",
      "50           0.063919          0.001142            0.058192           0.000242\n",
      "51           0.063922          0.001113            0.057974           0.000303\n",
      "52           0.063899          0.001144            0.057896           0.000275\n",
      "53           0.063926          0.001120            0.057826           0.000273\n",
      "54           0.063888          0.001130            0.057693           0.000273\n",
      "55           0.063869          0.001125            0.057610           0.000279\n",
      "56           0.063860          0.001138            0.057496           0.000299\n",
      "logloss of train :\n",
      "0.05812395436573874\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEXCAYAAABCjVgAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xt8XAWd9/HPb2aStE3SpE3Tkt4vFGhBRG0LeEEEXAGVIiIK1BfghXVf8uBlfVj08XH3wcX7BUXWBVfAXUVEVEDAReSqLNQWKFAutVfa0Gvapk2aJs3M/J4/zpn0NEyaNM3kTDLf9+t1XjPnMuf8zpnkfOfczd0REREBSMRdgIiIFA+FgoiIdFEoiIhIF4WCiIh0USiIiEgXhYKIiHRRKIhEmNmXzOw/4q5DJC4KhSHGzKrMbJ2ZXRTpVm1m683s/Ei3eWZ2r5ntNLNmM3vJzK41szFh/0vNLGNmrWGzxsz+ocC1n2pmjYWcxqHIV4+7f83dP1Gg6a0zszMKMe5CGKzva6gtl+FOoTDEuHsrcDnwAzOrDzt/C1jq7ncCmNlbgUeBJ4Bj3L0WOBNIA2+MjO5Jd69y9yrgfOBbZvamwZkTORRmloq7BikR7q5mCDbArcAvgVOB7UBDpN9fgOt7+fylwF+6dfsrcFGk/RzgRaCZIGTmRPrNCbs1h8OcE+l3NvAS0AK8BnwBqAT2AlmgNWwm9jBfNwD3hZ9fDMzqw/I4BngQ2AGsAC7oTz3AvwA/Dz83HXDgMmADsBP4FDAfeD6c9x9FpjMLeDj8PpqAXwC1Yb//Cqe1N5zWVX1YxuuAfwqn1QGkwvbXwnlZAZyeZ1mcBGwGkpFuHwCeD98vAJYCu4EtwPd6WKanAo099KsB/hPYBrwKfBlIhP2SwHfDZbAWuCJcjqkexrUOOKOHfp8EVoXf6z25vxnAgO8DW4Fd4TI6rqfvO+7/16HUxF6Amn5+cTAG2BT+410W6V4JZIBTe/n8pURCIVzRNQNHhe1HAXuAdwNlwFXhP2d52L4K+FLYflr4D3h0+NlNwDsidb45fN/jSiZSx63hCmBBuBL8BXB7L5+pJFhpXxZ+5s3hcjn2UOshfyj8OzAC+DugHbgLGA9MCldK7wyHPzJcXhVAPfA4cF1k3Aes/A62jCPDLwOmACOBo8P5nBipL29gAquBd0fafw1cHb5/Evho+L4KOKmHcfT4fREEwt1AdVjH34CPh/0+RbBSnhwu7z/Rj1AI/66awu+zArgeeDzs9x7gaaCWICDmEP4w6un7VtO3RruPhih330nwC3MU8NtIrzEEuwU35zqY2bfC4wp7zOzLkWFPCru3Emwl/BewMuz3YeA+d3/Q3TuB7xCsmN5K8Eu0CviGu+9z94eBe4ELw892AnPNbLS773T3Zw5x9n7r7n919zRBKJzQy/DvA9a5+y3ung6n9xuCXWIDUc9X3b3d3f9IsBL/pbtvdffXgD8DbwJw91Xh8upw923A94B3HmS8B1vGOT909w3uvpcg7CvCeSlz93XuvrqHcf+S8Psws2qCX8+/jCyPI81snLu3uvtTh7IwzCwZ1v5Fd29x93UEWwYfDQe5APiBuzeGf6ffOJTxR1wM3Ozuz7h7B/BF4GQzmx7OQzXBFqK5+8vuvikyf4fzfZc0hcIQZWaLCH6h/Qn4ZqTXToLdFA25Du5+lQfHFX5H8Es65yl3r/XgmMIRwLHA18J+Ewl2C+TGkSX4lTop7Lch7JbzatgP4IMEK6FXzewxMzv5EGdvc+R9G0EAHcw04MQw4JrNrJlghXLEANWzJfJ+b572KgAzG29mt5vZa2a2G/g5MO4g4z3YMs7ZEOm/CvgswdbM1nBaE3sY923AeWZWAZwHPOPuuWl9nGAr5RUzW2Jm7ztIjfmMI9hCfDXSLfr9T4zW3e39oei+fFoJds1NCn+I/IhgV+MWM7vJzEaHgx7u913SFApDkJmNJ9if+kng74ELzOwUAHffQ7Af/rxDGae7byH4df3+sNNGgpVtbppGsBvjtbDfFDOL/v1MDfvh7kvcfSHBLpa7gDtykzmUmg7BBuCxMOByTZW7/8Mg1/P1cJzHu/toYBHBro2c7tM72DLO+xl3v83d3x5+zjnwB0F0uJcIVqhnARcRhESu30p3v5BgeXwTuNPMKvs+mzQR/BqfFunW9f0T7L6ZHOk35RDGHdV9+VQCdez/O/uhu7+F4MfMUcD/Drv39H1LHygUhqYfAXe5+yPhJvNVwE/CX4WE7R8zs6vDAMHMJgMzehqhmdURHIx8Mex0B/BeMzvdzMqAfyQ42Pk/BKGzB7jKzMrM7FSCMLndzMrN7GIzqwl3iewm2O0BwS/sOjOrGaDlkHMvcJSZfTSsp8zM5pvZnEGup5rgIHKzmU0iXElFbAFmRtoPtoxfx8yONrPTwu+5nWArJZNv2NBtwJXAKQTHFHLjWWRm9eGWSXPYucfxmNmIaEOwJXoHcG14OvQ04PMEW0a5+fqMmU0ys1qCg+O9Kes2nVRY/2VmdkI4z18DFrv7uvD7PTFcbnvC5ZHp5fuWvoj7oIaaQ2uAcwl+QdV26/4QcG2k/UTgfoJ/+mZgOXAtUBf2v5TgnyV35s1Wgn3O4yPj+ADBAcNdwGOEB27DfseG3XaFw3wg7F4O/DfBbqzdwBLg7ZHP3UywC6CZns8++tdI+6n0cnA6HO5ogjOWtoXjf5jgWMQh1UP+A82pyPCNRA7iE6wIvxxZJk+Hy3MZwUq+MTLsQmB9OK0v9GEZr+PAA9PHExz7aSE4GH9vvmUYGX4qwQr8vm7dfx5+360EPwLO7eHzp4bz3705kuDY1c/D5b0B+Ar7zz5KEWzJbic4++hzBFsW1sN01uWZxr+G/T5FcNA8N7+Tw+6nE5xx1Mr+M72qevu+1fTeWLiARUQKwszOAv7d3af1OrDETruPRGRAmdlIMzvbzFLhbrR/JjjJQYYAbSnIkGBm7wD+kK+fB2dPSZEws1EEu8KOITjucR/wGXffHWth0icKBRER6aLdRyIi0mXI3WRr3LhxPn369LjLEBEZUp5++ukmd6/vbbghFwrTp09n6dKlcZchIjKkmNmrvQ+l3UciIhKhUBARkS4KBRER6aJQEBGRLgoFERHpolAQEZEuCgUREelSMqGw7pVnWfy76+MuQ0SkqJVMKGxeehcnPvdldjdvj7sUEZGiVTKhUDYmeCLg9o1rYq5ERKR4lUwoVNYHz/fYvaVPV3qLiJSkkgmFMUdMB6B9+/p4CxERKWIlEwp1DdPIuOHNjXGXIiJStEomFFJl5Wy3MSRbN8ZdiohI0SqZUADYmRrPyL2b4i5DRKRoFTQUzOxMM1thZqvM7Oo8/S81s21mtixsPlHIevaMmMDofdsKOQkRkSGtYA/ZMbMkcAPwbqARWGJm97j7S90G/ZW7X1GoOqL2VTYwruUpPJvFEiW1kSQi0ieFXDMuAFa5+xp33wfcDiws4PR6N3oSo6yDXTu0tSAikk8hQ2ESsCHS3hh26+6DZva8md1pZlPyjcjMLjezpWa2dNu2/q/QK8aGF7BtWtvvcYiIDGeFDAXL0827tf8emO7uxwN/An6Wb0TufpO7z3P3efX1vT53ukejwgvYWrau6/c4RESGs0KGQiMQ/eU/GTjgfFB33+7uHWHrT4C3FLAexk6cCUD79g29DCkiUpoKGQpLgNlmNsPMyoGPAPdEBzCzhkjrOcDLBayHseMn0+lJXcAmItKDgp195O5pM7sCeABIAje7+4tmdg2w1N3vAa40s3OANLADuLRQ9QAkUym22VhSe3QBm4hIPgULBQB3vx+4v1u3r0TefxH4YiFr6K65rJ5RezcP5iRFRIaMkjtZf8+II6jp3Bp3GSIiRankQqGzsoH67HY8m427FBGRolNyoWA1k6iwTnY06R5IIiLdlVwolNdNBWDHRl3AJiLSXcmFQrUuYBMR6VHJhUJtwwwAOnfoAjYRke5KLhTG1k9knyfJ7tIFbCIi3ZVcKCSSSZoS4yhr1bUKIiLdlVwoADSXjWdUu0JBRKS7kgyFthFHUJvWBWwiIt2VZCh0VjUwLrudbCYTdykiIkWlJEMhUTOJcsuwY4sONouIRJVkKFSMDS5g2755XbyFiIgUmZIMheoJwQVse3QBm4jIAUoyFOoagiew7dMFbCIiByjJUKipm0C7l+G6gE1E5AAlGQqWSNCUGEf5Ht0pVUQkqiRDAWBX2XgqO7bEXYaISFEp2VBoG3kEtZ3b4i5DRKSolGwoZKonMs53kO7sjLsUEZGiUbKhYDWTSVmWps3r4y5FRKRolGwojKybAkDzZj2BTUQkp2RDoXrCdAD2bHs13kJERIpIyYbC2InBBWydO3StgohITsmGwuiaOtq8Ana/FncpIiJFo2RDwRIJmpK6gE1EJKpkQwFgd9l4qjr0BDYRkZySDoW9IxsYk9YFbCIiOSUdCpnqidR5M/s6OuIuRUSkKJR0KCRrJ5Mwp2nTurhLEREpCiUdChVdF7Cti7cQEZEiUdKhUBNewNbWpAvYRESgxEOhblLuAjbd/0hEBAocCmZ2ppmtMLNVZnb1QYY738zczOYVsp7uqkaPZTs1JHesHszJiogUrYKFgpklgRuAs4C5wIVmNjfPcNXAlcDiQtVyMFvKpzJ6j26KJyIChd1SWACscvc17r4PuB1YmGe4rwLfAtoLWEuPWqtn0tC5Hs9m45i8iEhRKWQoTAI2RNobw25dzOxNwBR3v7eAdRyU182mhj1s37YxrhJERIpGIUPB8nTzrp5mCeD7wD/2OiKzy81sqZkt3bZtYK9AHjUx2KO1efULAzpeEZGhqJCh0AhMibRPBqI/x6uB44BHzWwdcBJwT76Dze5+k7vPc/d59fX1A1rk+JlvAGBP44sDOl4RkaGokKGwBJhtZjPMrBz4CHBPrqe773L3ce4+3d2nA08B57j70gLW9Dr1k2bR5hVkm/42mJMVESlKBQsFd08DVwAPAC8Dd7j7i2Z2jZmdU6jpHqpEMsnG1GRG7dZpqSIiqUKO3N3vB+7v1u0rPQx7aiFrOZhdldOZuPv5uCYvIlI0SvqK5pzOMbNpYBt7WnfHXYqISKwUCkDFEccAsElnIIlIiVMoAGOmHQvAzvU6A0lESptCAWiYeSwZN9JbVsRdiohIrBQKQMWISjYnJlDevDLuUkREYqVQCDWNmM6YtnVxlyEiEiuFQmhv7SwmZTaS7uyMuxQRkdgoFELJ+qOosE42rdeVzSJSuhQKodFTjgOgaa1OSxWR0qVQCDXMOh6A9k0vx1yJiEh8FAqh0WPHs50aEtt1BpKIlC6FQkTwaM41cZchIhIbhUJEa/VMJurRnCJSwhQKEblHczbp0ZwiUqIUChGVk+YAsEU3xhOREqVQiKifEZyB1PraSzFXIiISD4VCxPjJwaM5fZtujCcipUmhEGGJJJv0aE4RKWEKhW6aK6dT374+7jJERGKhUOgmPeZIjnA9mlNESpNCoZvyI+aQMGfj6uVxlyIiMuh6DQUzm2VmFeH7U83sSjOrLXxp8RgbPpqzeb1CQURKT1+2FH4DZMzsSOCnwAzgtoJWFaOGmceRcaNTj+YUkRLUl1DIunsa+ABwnbt/DmgobFnxKR8xKnw056q4SxERGXR9CYVOM7sQuAS4N+xWVriS4tc0Yhpj29bGXYaIyKDrSyhcBpwMXOvua81sBvDzwpYVr/YxRzM508jetj1xlyIiMqh6DQV3f8ndr3T3X5rZGKDa3b8xCLXFZsT0+ZRbhrXLn4q7FBGRQdWXs48eNbPRZjYWeA64xcy+V/jS4jP5uFMAaF75ZMyViIgMrr7sPqpx993AecAt7v4W4IzClhWvuonT2UodZZufibsUEZFB1ZdQSJlZA3AB+w80D3uvVc2lofXFuMsQERlUfQmFa4AHgNXuvsTMZgLD/kHGnRPexGTfTNMWPXBHREpHXw40/9rdj3f3fwjb17j7BwtfWrxGH3kyABuW/znmSkREBk9fDjRPNrPfmdlWM9tiZr8xs8mDUVycpr3hbWTcaFv717hLEREZNH3ZfXQLcA8wEZgE/D7sNqyNrKphQ2oqVduWxV2KiMig6Uso1Lv7Le6eDptbgfq+jNzMzjSzFWa2ysyuztP/U2b2gpktM7O/mNncQ6y/oJpq3sC09lfIZrJxlyIiMij6EgpNZrbIzJJhswjY3tuHzCwJ3ACcBcwFLsyz0r/N3d/g7icA3wKK6/qHSfOotVY2rNEdU0WkNPQlFD5GcDrqZmATcD7BrS96swBYFR6Y3gfcDiyMDhBe/5BTCXhfih4s9XPeCsDmF5+IuRIRkcGR6m0Ad18PnBPtZmafBa7r5aOTgA2R9kbgxO4Dmdmngc8D5cBp+UZkZpcDlwNMnTq1t5IHzJSj3kybV5BpXDpo0xQRiVN/n7z2+T4MY3m6vW5LwN1vcPdZwD8BX843Ine/yd3nufu8+vo+Hc4YEIlUGa9WHEVd8/ODNk0RkTj1NxTyrfC7awSmRNonAwe7Eux24Nx+1lMwu+veyIzONbTvbYu7FBGRgutvKPRl3/8SYLaZzTCzcuAjBKe2djGz2ZHW91KEV0oHd0xN646pIlISejymYGYt5F/5GzCytxG7e9rMriC4RUYSuNndXzSza4Cl7n4PcIWZnQF0AjsJHuRTVCYd9w54EnaufBLm5z3kISIybPQYCu5efbgjd/f7gfu7dftK5P1nDncahTZu4kyaGENq07NxlyIiUnD93X1UOsxorDyWhlZdqyAiw59CoQ/2TTiBKb6J7ds2x12KiEhBKRT6oDq8Y+r653XHVBEZ3hQKfTD9DW8jqzumikgJ6Muts1vMbHe3ZkN4O+2Zg1Fk3EZWj2FDcgqVTbpjqogMb73e5oLgJnUbgdsITkf9CHAEsAK4GTi1UMUVk201xzFr55/JZrIkktrAEpHhqS9rtzPd/UZ3b3H33e5+E3C2u/8KGFPg+orHpHmMoYX1a16OuxIRkYLpSyhkzewCM0uEzQWRfkV1V9NCqp/zNgBuu/OOmCsRESmcvoTCxcBHga1h81FgkZmNBK4oYG1FZeqc+exkNKeX63oFERm++nLr7DXA+3vo/ZeBLad4WSLJ2poTOXLXYtLpNKlUXw7HiIgMLX05+2hyeKbRVjPbYma/MbPJg1FcsUkc9W7q2M2KZSWThSJSYvqy++gWgrubTiR4cM7vw24lZ+ZJ55B1Y+dz9/c+sIjIENSXUKh391vcPR02twKD96SbIjK6roE1ZUcydtPjcZciIlIQfQmFJjNbZGbJsFkEbC90YcVqx8R3cnTnKzRt2xJ3KSIiA64vofAx4AJgM7AJOB+4rJBFFbO6E84mac7qp34fdykiIgOu11Bw9/Xufo6717v7eHc/FzhvEGorSjOOP4VdVOEr/xR3KSIiA66/92v4/IBWMYQkUmWsGT2fWbufIpPJxl2OiMiA6m8o2IBWMdTMOp16drLy+SfjrkREZED1NxRK5vYW+cw4aSEATTo1VUSGmR4vyzWzFvKv/A0YWbCKhoDaCVNZm5xJ7WuPxV2KiMiA6nFLwd2r3X10nqba3Uv+Hg9NDe/g6H0vsXNHyZ6dKyLDkB4M0E+1bziLMsuwcvF9cZciIjJgFAr9NPPNp9PKSDIr/hh3KSIiA0ah0E/JsnJWV81jevOTZHVqqogMEwqFw5CZeRoNNLH65afjLkVEZEAoFA7DtBODU1O3PKPjCiIyPCgUDkPdpFmsT05l9IaH4y5FRGRAKBQO09bJ7+HYfc+zYd2quEsRETlsCoXDNO20j5M0Z+0jJfncIREZZhQKh6l+2hz+Vj6XKevv1llIIjLkKRQGwJ45FzDDN/DiM3oim4gMbQqFAXDM6ZfQ4WXsevI/4y5FROSwKBQGwMjRY3ml5u3M3f5H2va2xV2OiEi/FTQUzOxMM1thZqvM7Oo8/T9vZi+Z2fNm9pCZTStkPYU0Yt7FjLUWnn/kzrhLERHpt4KFgpklgRuAs4C5wIVmNrfbYM8C89z9eOBO4FuFqqfQZr/1XHZQQ/KFX8VdiohIvxVyS2EBsMrd17j7PuB2YGF0AHd/xN1z+1ueAiYXsJ6CSqTKWDvxvbyx7Uk2b34t7nJERPqlkKEwCdgQaW8Mu/Xk48Af8vUws8vNbKmZLd22bdsAljiwGk65jHLLsOKhn8VdiohIvxQyFPI9xznvYzzNbBEwD/h2vv7ufpO7z3P3efX19QNY4sCaeMwC1qVmMH71b3Ev6SeWisgQVchQaASmRNonAxu7D2RmZwD/BzjH3TsKWM+g2HHkB5mTXcmK5bpzqogMPYUMhSXAbDObYWblwEeAe6IDmNmbgBsJAmFrAWsZNLPPuIy0J9j6l1vjLkVE5JAVLBTcPQ1cATwAvAzc4e4vmtk1ZnZOONi3gSrg12a2zMzu6WF0Q0b1uMmsqFrAUVvup6OzM+5yREQOSaqQI3f3+4H7u3X7SuT9GYWcflwskeQItvPEg7/ibWcvirscEZE+0xXNBTDnM3exxcZT8/T1ukmeiAwpCoUCsFQ5W4+/nOMyr7Dk8XvjLkdEpM8UCgUy9+xPs4MaUv/zfZ2eKiJDhkKhQJIVo9hwzGW8pfMZnln8aNzliIj0iUKhgOa8/3O0MIrOR78TdykiIn2iUCig8spaVs+4mAV7n+CFZX+NuxwRkV4pFArs6IVX0WHl7HpwyN4AVkRKiEKhwEbWjmfFpA9yUutDrHhledzliIgclEJhEMxaeDVuxqY/aGtBRIqbQmEQVI+fxsvj38fJzfezdt3quMsREemRQmGQTHn/F0mRZs3d34i7FBGRHikUBsmYKXN4ecL7OGXHb3hm6ZNxlyMikpdCYRAdedF32WsjSdz/BTo603GXIyLyOgqFQTSidgKbF1zNCdnlPPbrH8VdjojI6ygUBtlRZ36atSPm8pYV32V942txlyMicgCFwmBLJBh9/vXU0sLKX16lm+WJSFFRKMSg7sh5vDLtIt7Veh9PPPZA3OWIiHRRKMTkmAu/zs7EGMY9ejUtbXvjLkdEBFAoxCY5soaWU7/KMazlL7fp2gURKQ4KhRhNP+VinrI38vYNN7L02WfiLkdERKEQKzOO/9QtYAmq7r6M15p2xF2RiJQ4hULMRk2YRet7/41jWMcrP/kE7ft0UZuIxEehUAQa5p/Lqrmf5vSOh/jDz74edzkiUsIUCkXiyPO/ypqakzm78ToefPC+uMsRkRKlUCgWiSTTLr+N3ak6jv3L/+LFVWvirkhESpBCoYgkK8dSfvEvqLPdtN12KTtadP2CiAwuhUKRqZk5n23v+Brzs8/x39+5hF1t++IuSURKiEKhCE0+/XJePepjXGQP8NgPP8GuPQoGERkcCoUiNe3C77F+9iWc0343j16vYBCRwaFQKFZmTL3oB6yffQkL2+/mkes/qWAQkYJTKBSzMBg2zP4o57bfxcPXX65gEJGCUigUOzOmXHQ9G2Yv4gPtv+OhH36S13buibsqERmmFApDgRlTLvoRjbM/ynkdd7H6B+/nqZfWxl2ViAxDCoWhwozJF11P0ynX8laWMeH2M7n9vgf15DYRGVAFDQUzO9PMVpjZKjO7Ok//U8zsGTNLm9n5haxlWDBj3GlX0LnoburKOnjfXxfxkxuvo7VDN9ETkYFRsFAwsyRwA3AWMBe40MzmdhtsPXApcFuh6hiORh75DqqvfII9NbO5fPO/cM93/57lG3TbbRE5fIXcUlgArHL3Ne6+D7gdWBgdwN3XufvzQLaAdQxLVjOJCVc+xOYjP8xF++4k85PTufFXd2mrQUQOSyFDYRKwIdLeGHY7ZGZ2uZktNbOl27ZtG5DihoVUBUcsuom299/ErPKdfPyly7jrmx/nwWWrdaxBRPqlkKFgebr1a03l7je5+zx3n1dfX3+YZQ0/o97yYao+/wzNR1/AouxdHPPb93Ddj/+NDTva4i5NRIaYQoZCIzAl0j4Z2FjA6ZW2UWMZd9GNpC+5j6qqKj639Uss//5Cvv+Lu1i/XeEgIn1TyFBYAsw2sxlmVg58BLingNMTIDXj7Yz5/GJaTr6Kd5Ut53MrL+GF687l2//5G1Zva427PBEpclbIfc9mdjZwHZAEbnb3a83sGmCpu99jZvOB3wFjgHZgs7sfe7Bxzps3z5cuXVqwmoeVth20PvZDypbcSEW2jT9kFvDszMs549TTmT99DGb59vCJyHBkZk+7+7xehxtqByQVCv3QtoO2x68nseRGRmT28FR2Dg+NfA918z/EwvmzaKgZGXeFIlJgCgV5vb072bf4p+z7661UtW1gl4/iruzbWTnpPN684B286+jxjKksj7tKESkAhYL0LJuFV59gz5M/pWLlfaR8Hy9np/Cn7FtonPAuph/3dk6fO4HZ46u0i0lkmFAoSN+07SD7/B20LfstozYvIUGWTT6WP2XezLKRJ1E26228ceZk5k8fw8xxVSQSCgmRoUihIIduz3ZY+QDty+8lueoBykiTJskL2Rkszs5hedlxMPUkjp46ibkTRzN34miOGD1CWxMiQ4BCQQ5PZzu8+gS+7gk6Vj9O2eZnSXqaDAlWZiex3GfwQnYGr5YfiR1xPDMmjmdGfSUz6iqZUV9Jw+gR2qoQKSIKBRlY+9qgcQm8+gTpxmfwjc9StrcJgCzGOm9gZXYiq3wiq7MTWZ+YQnrsLMaOHUdDzQgm1o5kYu0IGmpGcsToEdRVlVNVkdJWhsgg6WsopAajGBkGykfBzHfCzHcGfzTu0LIZNi0jsek5Zmx+galbV/DunctIeHhTvt2ws6WWDV7Pusw4Nng9i308r/k4tvgYdiXHkqqso666grGV5YwZVU7NyDJqR5VRO7KM2lHljB6ZoqqijMqKJNUVZVSNSFFVkaI8pUeBiBSCQkH6xwxGNwTN0WdhhH9MmU7YsRaaVkDT3xizcx1jdr7KcTtfxXYvwbIH3sU1va+M5p1jaWoeQ5NXszVTxZZ0FY0+mue9mp1Us8sr2UUlzV7FLipJk6IsaYwsS1JZkWJUefA6oizJyFxTnmREWYKKVJKKsgQVyQQVZUnKkwnKUwnKkglSSaMsaaQSQXtZ0kglE5S5GvPiAAAKr0lEQVQlgtdU0ihLHDhcKvKaNCOZe02EjZl2m8mQplCQgZUsg/qjgiYiAZBJQ8tG2NUYbGW0biHVsplxLZsZ17o5ONDd1ojvacKynT1OYl9iJB3JKtqTlbRZJXsyo2hpG0Wrj2SPV9DiFbRkR7A7U86uTDk7suW0ZMpop5x2L2cvFXRQRruX00FZV5MmSf77OB66hEEyYaSzjgGV5SkSYXAkzEgmIGmGmZFIQMKC7kZQQq4KMyNhYBi5PW0JC95H97yFnwy7B59JdPus2f7PJsy6xm+5z0Xac+PqGnsPi2X/dOyA8R8wrrAGz90PM7rHOuyXsFwN0XnbP0/727rX1nd2kPFFOcGGcPDeOZQ97NG6PM/sRqcbnd/u/YL+r6/wvcc3MH/62L4X1A8KBRk8yRTUTg2agzB36NgNe5pg707Y2wztzV3vy/fupLxjF9UdLdC+Oxi2fSPsa4V9e6Cjha5/RSO4yUqy9/Icw5MVZJPlZBMVZJLlZK2MTKKcbCJF1spIWxlZS4WvSdKWImNlZEiSsRRpkmQtRYYkaZJkSJAhRdqSpAn6739NkA2HSZMIPutG1hJkSOIkyWBkPEHGcuMKXz1B1pLBdCzV9b6TBBlP0hlOv9OTpD2Yt2wWMmTJOmQ9WNk5gHvXijC34u5aoXnPtzbOHY/MuneN87WdewGYMHpE1wp1a0sHONRXV+xfIRts3d0BwLiqCjwcx/Y9QbfaUeVd09+1dx/uUDuq7IDadu0NfjjUjCzrqilft2j36hGp8LuG1vY0HumW09oebM2ODsdhkc/nagBobjuwW766cvPbva7cct0dqSu3nHN1VVV0q6sjzeN/28bDXziVQlIoSPExgxE1QdMf7tC5NwiIfS3BmVTpvUG3zr3Q2QbpDki3H/Bq6XYs3UEisy/SrwMy+4LdYrnXbCekWyCb3t+eCZtsOmjPZvb38yJ4hpQlIJECS0IibCzfayJ4tcT+JpEMf9YmI+25z3Rrb0i+fvgDxpPo27iin+tqyN89keo2rlS36UXfE3lfsf89FqktceCyiI43kQy2hhOp/U3uM13jCMeD5enXbROvCCkUZPgxCw6Ml48CiuD5G9ns/rDIBUemEzwTds+ETTrsltn/esD7dBAw2XSkyUTCKX1gkwnDKRdSXdPL5plWNtKezd9Ea8l9JpsOgjPa3T3PeHLt0X75pp/eP5xn6ecjWIqbRcMmkSdI8gVp2P+0/wvHX1DQ8hQKIoWWSECiHNB9pQ6Ze9iEYYN3C6nc+/SBwXdAAGW6BVH2wFDqGmdk3HlDMEPe0I3W9LpaI6/dx9v1PhKAB4wje+B4sxmomlDwRa5QEJHi1bW7RacgDxYtaRER6aJQEBGRLgoFERHpolAQEZEuCgUREemiUBARkS4KBRER6aJQEBGRLkPuITtmtg14tZ8fHwc0DWA5xWS4zpvma+gZrvM21Odrmrv3et+XIRcKh8PMlvblyUND0XCdN83X0DNc5224zld32n0kIiJdFAoiItKl1ELhprgLKKDhOm+ar6FnuM7bcJ2vA5TUMQURETm4UttSEBGRg1AoiIhIl5IJBTM708xWmNkqM7s67nr6y8xuNrOtZrY80m2smT1oZivD1zFx1tgfZjbFzB4xs5fN7EUz+0zYfTjM2wgz+6uZPRfO2/8Lu88ws8XhvP3KzIbko9nMLGlmz5rZvWH7cJmvdWb2gpktM7OlYbch//fYm5IIBTNLAjcAZwFzgQvNbG68VfXbrcCZ3bpdDTzk7rOBh8L2oSYN/KO7zwFOAj4dfkfDYd46gNPc/Y3ACcCZZnYS8E3g++G87QQ+HmONh+MzwMuR9uEyXwDvcvcTItcnDIe/x4MqiVAAFgCr3H2Nu+8DbgcWxlxTv7j748CObp0XAj8L3/8MOHdQixoA7r7J3Z8J37cQrGQmMTzmzd29NWwtCxsHTgPuDLsPyXkzs8nAe4H/CNuNYTBfBzHk/x57UyqhMAnYEGlvDLsNFxPcfRMEK1dgfMz1HBYzmw68CVjMMJm3cBfLMmAr8CCwGmh293Q4yFD9m7wOuArIhu11DI/5giC4/2hmT5vZ5WG3YfH3eDCpuAsYJJanm87FLUJmVgX8Bvisu+8OfngOfe6eAU4ws1rgd8CcfIMNblWHx8zeB2x196fN7NRc5zyDDqn5inibu280s/HAg2b2StwFDYZS2VJoBKZE2icDG2OqpRC2mFkDQPi6NeZ6+sXMyggC4Rfu/tuw87CYtxx3bwYeJThuUmtmuR9mQ/Fv8m3AOWa2jmCX7GkEWw5Dfb4AcPeN4etWgiBfwDD7e8ynVEJhCTA7PCuiHPgIcE/MNQ2ke4BLwveXAHfHWEu/hPuifwq87O7fi/QaDvNWH24hYGYjgTMIjpk8ApwfDjbk5s3dv+juk919OsH/1MPufjFDfL4AzKzSzKpz74G/A5YzDP4ee1MyVzSb2dkEv2KSwM3ufm3MJfWLmf0SOJXgNr5bgH8G7gLuAKYC64EPuXv3g9FFzczeDvwZeIH9+6e/RHBcYajP2/EEByWTBD/E7nD3a8xsJsEv7LHAs8Aid++Ir9L+C3cffcHd3zcc5iuch9+FrSngNne/1szqGOJ/j70pmVAQEZHelcruIxER6QOFgoiIdFEoiIhIF4WCiIh0USiIiEgXhYKIiHRRKIj0gZmdEF7rkms/Z6BuwW5mnzWzUQMxLpHDpesURPrAzC4F5rn7FQUY97pw3E2H8JlkeD8lkQGlLQUZVsxsevignp+ED7T5Y3hriXzDzjKz/w7vgvlnMzsm7P4hM1sePhTn8fDWKNcAHw4fuPJhM7vUzH4UDn+rmf04fEjQGjN7pwUPQ3rZzG6NTO/HZra024N2rgQmAo+Y2SNhtwvDh7ssN7NvRj7fambXmNli4GQz+4aZvWRmz5vZdwqzRKXkuLsaNcOmAaYTPLDnhLD9DoLbLOQb9iFgdvj+RIJ790Bwq41J4fva8PVS4EeRz3a1Ezz46HaCO4QuBHYDbyD40fV0pJax4WuS4KZ4x4ft64Bx4fuJBLdPqCe4vcLDwLlhPwcuyI0LWMH+rf3auJe9muHRaEtBhqO17r4sfP80QVAcILxF91uBX4fPObgRaAh7PwHcamafJFiB98Xv3d0JAmWLu7/g7lngxcj0LzCzZwjuB3QswVMAu5sPPOru2zx4JsEvgFPCfhmCu8hCEDztwH+Y2XlAWx/rFDmoUnmegpSW6M3XMkC+3UcJgofBnNC9h7t/ysxOJHii2DIze90wB5lmttv0s0DKzGYAXwDmu/vOcLfSiDzjOdgDJNo9PI7g7mkzWwCcTnCH0isIbl0tcli0pSAlyd13A2vN7EMQ3LrbzN4Yvp/l7ovd/StAE8GzOFqA6sOY5GhgD7DLzCYQPC88JzruxcA7zWxc+GzxC4HHuo8s3NKpcff7gc8SPPtZ5LBpS0FK2cXAj83sywTPTb4deA74tpnNJvjV/lDYbT1wdbir6euHOiF3f87MniXYnbSGYBdVzk3AH8xsk7u/y8y+SPBMAgPud/d89+yvBu42sxHhcJ871JpE8tEpqSIi0kW7j0REpIt2H8mwZ2Y3EDxPOOoH7n5LHPWIFDPtPhIRkS7afSQiIl0UCiIi0kWhICIiXRQKIiLS5f8DKunpngm6mS4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "xgb7_4 = XGBClassifier(\n",
    "        learning_rate =0.2,\n",
    "        n_estimators=500,  #数值大没关系，cv会自动返回合适的n_estimators\n",
    "        max_depth=5,\n",
    "        min_child_weight=2,\n",
    "        gamma=0,\n",
    "        subsample=0.8,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel=0.7,\n",
    "        objective= 'binary:logistic',\n",
    "        reg_alpha=0.1,\n",
    "        reg_lambda=0.5,\n",
    "        seed=3)\n",
    "\n",
    "modelfit(xgb7_4, X_train, y_train, cv_folds = kfold)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从上面几组学习率来看，0.1的学习率比较好，log_loss最小"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEWCAYAAABi5jCmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzsnXd4VVXWh99FIoTQkV79QEpCFURQIUYRcAQUkVFRLCgqVhCVMnZ0BBQUGXEQRUGxDKjUYIdYQGYEQRQEQYh0Agk1tCSs7499EgOk3JSbG3LX+zz3yT3n7LP3XsCTzdm/81tLVBXDMAzDSKNEoCdgGIZhFC1sYTAMwzBOwhYGwzAM4yRsYTAMwzBOwhYGwzAM4yRsYTAMwzBOwhYGw/AREZkkIk8Eeh6G4W/EfAyGvxGROKA6kJrhdGNV3Z6PPqOB6apaJ3+zOzMRkanAVlV9PNBzMYof9sRgFBY9VbVshk+eF4WCQERCAzl+fhCRkEDPwSje2MJgBBQR6SAiS0Rkn4j87D0JpF3rLyK/ichBEdkoInd758sAnwK1ROSQ96klIlNF5LkM90eLyNYMx3EiMkxEVgFJIhLq3fexiOwWkU0i8mA2c03vP61vERkqIvEiskNEeonIlSLyu4gkisg/Mtz7tIh8JCL/8eL5SURaZbgeISKx3p/DahG56pRx/y0iC0QkCbgDuAkY6sU+z2s3XET+8PpfIyLXZOjjNhH5XkTGisheL9a/ZbheWUTeFpHt3vXZGa71EJGV3tyWiEhLn/+CjTMSWxiMgCEitYEY4DmgMvAI8LGIVPWaxAM9gPJAf+BlEWmjqknA34DteXgC6Qt0ByoCJ4B5wM9AbaAzMFhEuvnYVw0gzLv3SeANoB/QFugEPCkiDTK0vxqY6cX6PjBbRM4SkbO8eXwBVAMeAN4TkSYZ7r0R+CdQDngHeA94wYu9p9fmD2/cCsAzwHQRqZmhj/bAOqAK8AIwRUTEu/YuEA408+bwMoCItAHeAu4GzgZeB+aKSCkf/4yMMxBbGIzCYrb3P859Gf432g9YoKoLVPWEqn4JLAOuBFDVGFX9Qx3f4H5xdsrnPCao6hZVPQK0A6qq6khVPa6qG3G/3G/wsa9k4J+qmgx8iPuF+4qqHlTV1cBqIOP/rper6kde+5dwi0oH71MWGO3NYyEwH7eIpTFHVRd7f05HM5uMqs5U1e1em/8A64ELMjT5U1XfUNVUYBpQE6juLR5/Awaq6l5VTfb+vAHuBF5X1f+qaqqqTgOOeXM2iiln7D6rccbRS1W/OuVcfeDvItIzw7mzgEUA3lbHU0Bj3H9iwoFf8jmPLaeMX0tE9mU4FwJ852NfCd4vWYAj3s9dGa4fwf3CP21sVT3hbXPVSrumqicytP0T9ySS2bwzRURuAYYA53inyuIWqzR2Zhj/sPewUBb3BJOoqnsz6bY+cKuIPJDhXMkM8zaKIbYwGIFkC/Cuqt556gVvq+Jj4Bbc/5aTvSeNtK2PzF6nS8ItHmnUyKRNxvu2AJtUtVFeJp8H6qZ9EZESQB0gbQusroiUyLA41AN+z3DvqfGedCwi9XFPO52BH1Q1VURW8tefV3ZsASqLSEVV3ZfJtX+q6j996McoJthWkhFIpgM9RaSbiISISJgn6tbB/a+0FLAbSPGeHrpmuHcXcLaIVMhwbiVwpSek1gAG5zD+/4ADniBd2ptDcxFpV2ARnkxbEentvRE1GLclsxT4L25RG+ppDtFAT9z2VFbsAjLqF2Vwi8VucMI90NyXSanqDpyY/5qIVPLmEOVdfgMYKCLtxVFGRLqLSDkfYzbOQGxhMAKGqm7BCbL/wP1C2wI8CpRQ1YPAg8AMYC9OfJ2b4d61wAfARk+3qIUTUH8G4nB6xH9yGD8V9wu4NbAJ2AO8iRNv/cEc4HpcPDcDvb39/OPAVbh9/j3Aa8AtXoxZMQWITNNsVHUNMA74AbdotAAW52JuN+M0k7U40X8wgKouw+kMr3rz3gDclot+jTMQM7gZRiEgIk8D56pqv0DPxTBywp4YDMMwjJOwhcEwDMM4CdtKMgzDME7CnhgMwzCMkzgjfQwVK1bUc889N9DTKHSSkpIoU6ZMoKcRECx2iz3YKOjYly9fvkdVq+bcElDVM+7TuHFjDUYWLVoU6CkEDIs9OCnusaekpGjr1q21e/fuqqr61Vdf6XnnnaetWrXS5s2b6/r1609qP3PmTAX0xx9/zPVYwDL18XdsQLaSRORBcVkz94rIKi9z4zIR6RiI+RiGYQSCV155hYiIiPTje+65h/fee4+VK1fSuXNnnnsuPVkwBw8eZMKECbRv397v8wqUxnAvLlFaXaCVqrYGbseZiwzDMIo9W7duJSYmhgEDBqSfExEOHDgAuK2kWrX+Skn1xBNPMHToUMLCwvw+t0LXGERkEs7KPxd4S1Vf9i6lWfpz5EhyKucMj/HTDIsuD7dI4bYgjBssdou9+BA3ujsAgwcP5oUXXuDgwYPp1958802uvPJKSpcuTWhoKKtWrQJgxYoVbNmyhR49ejB27Fi/z7HQFwZVHSgiVwCXquoer5jIKFwO+O5Z3ScidwF3AVStWpUZVwSfIHXo0CGmBmHcUDxjj4+PZ9SoUSQmJiIi9OjRgz59+rBhwwZeeukljh8/TkhICMnV7mbqFW348ssv+fBDlz6pdOnSDB48mOL+EkZx/HuPjY3lhx9+IDk5mYMHD7Jy5UoSEhKIjY3lySef5NlnnyUyMpJ33nmHvn378vDDDzNkyBCGDx9ObGws+/btY/ny5Rw6dMh/k/RVjCjID5CAyxz5HhCNS362Cdjry/0mPgcfxTH27du36/Lly1VV9cCBA9qoUSNdvXq1dunSRRcsWKCqqjExMdqqVStVVV28eLEmJiaqquqCBQv0ggsuCMzEC5Hi+Peuqjp8+HCtXbu21q9fX6tXr66lS5fWK6+8Uhs0aJDe5sMPP9SIiAjdt2+fnn322Vq/fn2tX7++lipVSmvWrJlrAZqiLj7jKnLdANyHSxh2lar+Hy7TZZVs7zSMYkLNmjVp06YNAOXKlSMiIoJt27adtM+8f/9+zj77bAAuuugiKlWqBECHDh3YunVr5h0bRZ5Ro0axdetW4uLi+PDDD7nsssuYM2cO+/fv5/ffXbb1ZcuWERERQYUKFdizZw9xcXHExcXRoUMH5s6dy/nnn++3+QVKYwjFpVz+FPhEVTd7JQRDcU8ThhFUxMXFsWLFCtq3b8/48ePp1q0bjzzyCCdOnGDcuHGntZ8yZQp/+9vfMunJOFMJDQ3ljTfe4Nprr6VEiRKICJ988klA5hKQlBgikgJE4mrJRuJy7wO8rKpPZHFPusZQpUrVtk+Of6MwplqkqF4adh3JuV1xpLjF3qL2X5m9jxw5wqBBg+jXrx9RUVFMmDCBVq1acckll7Bo0SLmzJnD+PHj09uvWLGC8ePHM2HCBCpU8FeG8KLBoUOHKFu2bM4NiyEFHfull166XFV9e8zwdc+pID+4fPlVcDnel+LeSKqCq1HbOKf7TWMIPopj7Js3b9aoqCgtU6aMVq9eXcePH6+qqmXLltULLrhAW7VqpW3atNGwsDBVVZ0+fbqee+65WrJkSW3durWuXLkykNMvFIrj37uvFHTsnAEaQzlgCXAxrvTgz8BCXPH0VgGak2EUKiEhIZQrV44BAwawfv16Jk6cyJo1a1BVevfuzcqVK7n22mvxajNTunRpUlNTWbRoEc8//zx33XVXgCMwiiuBypVUHugCtAP+iavNezFOc3g8QHMyjEJl48aNxMTE0KJFC2JjY9m1axezZs0iMjKS119/nffff5+kpCSaNWsGwIIFC9i7dy/33nsvqamprFu3LsARGMWVQIvPDYGPgFXACeAQPojPZnALPopb7HGju9OxY8e0rVXi4uKIiorigQceoHfv3nTr1o3ExEROnDiRnhbhzTff5M03XXKAsWPHsnZtdpU/DSPvBEp8jgPOB6YCo1X1e+/818AwdXVmT70no8Gt7YwZMwptvkUFE+Jc7FkZww4cOMDIkSPZuXMnNWrU4KmnnqJcuXJF3hhm4nPW2L/54BKf0wxuR3FPCSu9zx9A25zuN/E5+MgYe1bGsEcffVRHjRqlqqqjRo3SoUOHqmrRNoYdP35cu3btquPGjUs/V758eT1x4oSqqp44cULDw8PTr/3888/aoEEDXbduXaHPNRDYv/mCgzNAfE4zuH0B/K6qrdUl0ksBtgdoTsYZQlbGsDlz5nDrrbcCcOuttzJ79myg6BrDVJU77riDiIgIhgwZkn6+Vq1afPPNNwAsXLiQ2rVrA7B582Z69+7Nu+++S+PGjQMyZyM4CLTG8D3QQdxrF+2B/aq6o7DnZJy5ZDSG7dq1i5o1awJu8YiPjz+tfVEyhi1evJh3332XFi1a0Lp1awCef/553njjDQYNGkRKSgphYWE8/PDDAIwcOZKEhATuvfdewBmili07bdfVMPJNoA1utYDPcK+sHgfuUNVMxQMzuBU/k1duSIs9O2NYjx49mD9/fvr1nj17Mm/evPTjM3Vv3vbZLfaC4EzQGOJwhrbyQFnv3JXAel/uN40h+MgYe1bGsIYNG2qnTp303HPPTf+peuYbw+zvPTgJRo0hzeD2NvCliBzDPUGcZUn0jJzIyhhWuXJlwsPDWb9+PeHh4VSuXBkwY5hh5JZAG9yOAeFAL1w1txJYEj0jB7IyhiUmJlKqVCkaNWpEjRo1SEhw/5TMGGYYuSPQ4vNvQFPgbNwicYP3yGMYWZKVMezFF19kw4YN6e3S3kQyY5hh5I6AGtxUdY93/DRwSFWzrFlnBreCFaPGjBnD0qVLqVixIm+//TYAkyZNYsmSJZx11lnUqlWLYcOGpY/3xx9/8NJLL5GUlESJEiWYNGkSJUuWzG6IAiWz2E18Lv5Y7MElPqdXcPOOJ+NSYvTx5X4Tn/PPN998o8uXL9dmzZqln/v88881OTlZVVWHDh2abhBLTk7WFi1apIu2e/bs0ZSUlAKbiy+cGntmxrDGjRvr9u3bVdWZ4DL+OzmTjWEmwAYnwSg+l8dtG90kIiE4vcE2fguRqKiodHE2ja5duxIa6nYXMxrBvvjiC1q2bEmrVi7x7dlnn01ISEjhTjgDmoUx7KqrrmLatGkATJs2jauvvhowY5hh5JaAagwi8gagwBqgWmHPxciat956i+uvvx6A33//HRGhW7du7N69mxtuuIGhQ4cGbG5ZGcOGDx/Oddddx5QpU6hXrx4zZ84EzBhmGLml0BcGVR0oIgNwbyKFA//FCc9hwOsi8oWqHsiuD8uumnfiRnfPsc0///lPQkNDuemmmwBISUnh+++/58cffyQ8PJzOnTvTtm1bOnfunK+55JWM4vOpfP3116edyyg+G4aRM4F6XXUrkIirxXCJqi4VkanA/KwWhVPEZ2ZcUaaw5hpwPvroI2JiYnjx/VSuuuoq+vTpw1tvvcXixYsRESpVqsSwYcOoUiVnC0hsbGz69507d5KUlHTSuc8++4x58+Yxbty49Hw9Bw4coEmTJvz6668AREREMHPmzELdTjp06NBJ8wwmLPbYQE8jIAQ0dl/FiIL88Jf4rLhUGMeBVNxi0Sun+4NJfP7ll1+0WbNmmpSUpF999ZV27txZf//9d92/f396m1deeUXvvvvuXPe9adOmk8TnTz/9VCMiIjQ+Pv6kdomJiXreeedpUlKSJicna+fOnXX+/Pl5DyoPmAgZnFjsBQdniviMq9pWXVVL4kp7Jqjq7ADNqUjy22+/0aFDB8LDwwkJCeGSSy5h1qxZlC9fPr1NUlJSevlHX+nbty8XXngh69ato06dOkyZMoX777+fgwcP0qVLF1q3bs3AgQMB5wcYMmQI7dq1o3Xr1rRp04bu3XPekjIM48wk0Aa3N1R1iXdpN9CmsOdT1GnevDmPPfYYCQkJHD16lAULFnD++e5V5Mcee4x33nmHChUqsGjRolz1+8EHH5x27o477siyfb9+/ejXr1/uJm8YxhlJkTC4eeceAZqq6oAs7iky2VVPnEjlxSceoWKls7n7kccZP3IEx466tKcHD+ynfsNG3PnQP/I9Tlom0ZiYGObMmUPJkiVp0KABpUqV4r777ktv995773H8+HH69++f7zGLKmZ0stiDjWJrcAMexKW92Abs569KbXuBKl6bK4DNOJ1hpC/9BlpjGDdunPbt21e7d+9+2rXevXvrtGnT/DLuokWLdMSIETpx4sSTzsfFxZ2kFRRHbK85OLHYCw6KkMZwLy6d9k3Ad/pXpbb9AJ65Lc3LcB5wtYhE+nlO+WLr1q3ExMQwYMDpDzYHDx5k4cKF9OrVq0DHTCs4s2vXLj755BP69u3L+vXr06/PnTuXpk2bFuiYhmEEL37TGDwtoQEwF3gri2Y9cQn0LlfV1SLyIXA1zvBWJBk8eDAvvPACBw8ePO3arFmz6Ny580nCcEFw7bXXkpCQwLFjx5g8eTKVKlViwIABrFu3jhIlSlC/fn0mTZpUoGMahhG8+G1hUGdkuwK4FGgOPC4iP+NqOp/lNbsfCAFe896qKQ8syKnvQBjc4kZ3Z/78+VSrVo22bdtm+n7xBx98kOmTRH757rvvAOdBiI6OBuDjjz8u8HEMwzDAz+JzmsiM0w9OqOohEbkSeEVVG4nI34Fu6gnOInIzcIGqPpBJXwHNrpqamkqfPn04fPgwlSpVYu/evSQnJxMeHk6nTp2488476d+/PzNnzvRb1lET4iz2YMNiD4z47O+FIQFnZvsFV9+5DfAY7knhfKAR8DROh1iGS4vxjqqOyq7fJk2aaGEXW3nppZdYtmwZBw4cYP78+YwePZrvv/+eefPmceONNwJQsmTJ9CRu/iDjE0OwYbFHB3oaAcFijy6w/kTE54XB3+JzmpHtSdwbSmM5uVLbj7jF4RlcdtXaOE2iSJGZ4NyhQwcARIQLLriAJUuW0Ldv30BN0TAMo8Dwt/icU6W2FBF5Cngd2AdsV9XV/ppTXslMcI6OjiY6Oprk5GTeffddpk+fTqdOnQI4S8MwjIKhMMTnKD29UtuSDE2vBjoB5YBHsurvFIMb/3pvjr+mnk6L2hX44YcfSE5O5uDBg6xcuZKEhISThOexY8fSoEEDUlNT/Z7wyhKKxQZ6GgHBYo8N9DQCQiBjLyyN4Ud1RXkmAwOA61T1IxHpAVwH1ACaAJWBFqoal12/haExHD16lKioKOLi4ti7dy9lypQhLCyMffv2ER4eTq1atQgNDaV+/frMmjWLEiX8n3bK9lujAz2NgGCxRwd6GgGh2GsMmnWltouB64GWQCncq6vZCs+FRalSpVi4cCHx8fEcPnyYJk2a8PjjjyMiLF26lMGDB5OQkMAVV1xRKIuCYRhGYeG332inVGp7CHgAZ1w7lKHZu7iniRo4kXqhqhYJBVdE0l8VS05OJjk5mRIlSlCiRAkaN27MwIEDOXHiBMOGDaN169aMHDkywDM2DMMoGPytMWRbqQ1oDOwTkU+AFkAJEQlR1dTs+va3wS2tyllqaipt27Zlw4YN3Hfffdxzzz2MGTOGZcuWkZKSwqBBg1i4cCErV67021wMwzAKG3+n3c62UpuIhOKE5/NwifT+A9wGTDm1I39WcDt+/DiDBg3i+PHjpKamcuuOGfTv358dO3Zw7Ngxypcvz+TJk2nYsCFDhw7l9ttvJzk5mfPPP5+jR48WmkBkQlxsoKcRECz22EBPIyAU2wpu5FCpDegArMLpDhtwC8PEnPot6OyqJ06c0IMHD6qq6vHjx/WCCy7QH374Qf/+97/rBx98oKqqbdu21Wuuueak+z7//HP9+9//XqBzyQ7LNBmcWOzBSXHOrppTpbbluLeRbgIiceJ0op/ndBqZ6Qn79u3jq6++ok+fPhw5coSUlBTi4uLSM50eO3aMMWPGpFc5MwzDKC4UivgMtFfVvd6l3bjXUsGlxfgFl311ObAJp0MUOqmpqbRu3Zpq1arRpUsXSpcuTVJSEm3atKFdu3Z07tyZ48eP8+KLLxIREUHLli3p2bMnl112WSCmaxiG4TcKJYmeZlGpTUT6AFfoyUn02qvq/Zn0VeAV3NIqpGXk0KFDPPHEE/Tv358xY8bw3nvvAa4mwvDhw3nrrawyiPsfSyhmsQcbFntgkugVas1nEbkUuAPomHYqk2aZrlSqOhmYDM7g9sBNVxfInLZs2cItt9zCzp07KVGiBHfddRe9evVi06ZNxMfHc//99/N///d/DBo0iEaNGgXUbGNmn+hATyMgWOzRgZ5GQAhk7P5eGMoBS0SkKm4LqT5OjI4Avse9tdRZRFbjtrUSgfl+ntNJhIaG8vjjj9O2bVtCQkI477zzqFChAgcOHODiiy9m4MCBHD58mGHDhnHHHXcU5tQMwzACgr8XhvI4Qfkw7hf+pTiD2wxcUr1QoCbQDFcXOgFX6rPQqFmzJrt37+bSSy8lNTWV+Ph4Lr74YmbNmsVnn31G3759iY+PJzEx0RYGwzCCgsLKrnoAl1n1NZzZra7XLAXYAnyGS4exF2eEy5aCMrilGdlatmzJihUriIuLIyoqin/961+sX7+eX3/9lf/973+89NJLPPXUU5QqVSrfYxqGYRR1Ck18FpFrcHmQqgHdVfUHr81YXGI9AV5V1cey6MsvFdzi4+MZNWoUe/bsIT4+nssuu4xhw4bx3XffMWbMGJKTkylXrhxHjx5l/vxC3eU6DRPiLPZgw2IPjPhcWAa397zjdjiD2yrv+CZgP87ktgo4Afwjp34L0uC2fft2Xbp0qXbt2lWff/55bdSoka5evVq7dOmiCxYsUFXVyZMna7ly5QpszLxiZp/gxGIPToq9wU3/yq46BvgcqC4iVXDlPseqaksgGjhK5m8q+Y0aNWowceJEIiIiGDFiBBEREWzbto3jx49z4MABTpw4wTvvvENERERhTsswDCNgFFZ21X/isqt+jNMVQnFPE5uBS7ycSdfjtIhV/ppTZixevJh3332XhQsXEhkZyaeffsr+/fu56KKL6NevH6VKlWLlypX85z//KcxpGYZhBAx/awwpuFQXtwKDgfW4p4QJqvqc9xTxGhAF1AO+UNVrsuirQA1up5rbjhw5wqBBg+jXrx9RUVFMmDCBVq1acckll7Bo0SLmz5/PuHHj8jVmfrH9Vos92LDYi6fGEAdUAWYCHbxzU4E+p7SrifM5nOVLvwWhMWzevFmjo6O1adOmGhERoU2aNNFx48apquqECRO0RIkSGhERoY8++qieOHHCNIYAY7EHJxZ7wUEuNAZ/vq76IO7p4G2cl6GPiOwFSgJXikgFoB+urGdFYL2qJvtrPqcSGhrKuHHjOO+887jxxhv57LPPuOKKK1i0aBFz5syhUaNGvPbaa0RGRrJw4UIaNWpUWFMzDMMIKP40uN0LxAOP4hzQvXA+heY4s9tiYIWq/iQi/wPqikikqq7x45zSqVmzJjVr1uT777/nww8/pFy5cvTs2ZOEhAQeffRRLrnkEgYNGkRKSgphYWFMnjy5MKZlGIYRcPyyMHjCcwPvsI86PaF7xjaqugPYISLnALWBZd7PHBeG/Brc0oxtAB07dmTTpk1ERUWxYsUKoqKiOH78OA8//DBly5Zl7NixtGvXLs9jGYZhnGn4TXw+NbOqiDwNHFLVsZm0PQf4Fmiuqgey6K9ADG5phrbExEREhK5duxIbG0u1atXYvHkzO3fupEqVKrz++uts3bqVkSNH8v777yNSqG/RZooJcRZ7sGGxFzPxGSc8jwB+A94DngbG4wxufbw29YEVuFxKm4GBvvSdH/F5+/btunz5clVVTUhI0PDwcH300Uf1888/1+TkZO3WrZvecMMNOnToUFVVbdCggcbHx+d5vILEhLjgxGIPToqzwe124EpVvQlnXOuOM7ilsdv7PI57rXW4iNTy54Rq1qxJmzZtUFUGDx5MjRo16NKlC127diU0NJRevXqRmprK1q1b+f333zl+/DhVqlTx55QMwzCKFP4Un88GSgFzReQtoD2uWtsBAHF7M5OANar6koicjR8Nd6eSZmw766yzePjhhylRogTPP/88t99+O8899xwAN9xwA9OmTSsS20iGYRiFhb81hhLAFcBXuOR5h4CzgIG4Mp7fAWuBc3CLyCRVvTeL/vJlcMvJ0JbG9OnTWbduHSNHjixyC4Ltt1rswYbFXsw0Bs2Fwc07Xwv4H1A9p37zozFs3rxZo6KitEyZMlq9enUdP368qqo+8sgjWqNGDS1durT26NFD9+7dm+cx/IXttwYnFntwUpw1hjTOBz70niL6AG+LyBYRiRGRWSKyCpiNK9bTyZ8TCQkJoVy5cgwYMID169czceJE1qxZQ8WKFalUqRJ//vknkZGRjBo1yp/TMAzDKLIUSs1nVf2/tO8iMhXnhI4ChgJ7VPUaEbkAiAWe9OdcNm7cSExMDC1atCA2NpZdu3Yxa9Ys3n77bY4dO0aXLl3Yv38/oaGhjBkzxp9TMQzDKJIUysJwChfi9Ia5wLnANhG5CvfW0iGcWzpb8mJwSzO1dezYMW3rKr1i2wMPPMBjj/1VH6hnz55cf/31uerfMAyjuODX7KpZDuqZ34AhQJiqDvGeGJYA7VV1eSb35NvgltHcpqocPXqU+++/nxMnTjB16lQ2b95Mz549SUxMNPG5iGGxW+zBRrEUn4EHcea2j4EfgGPAI3qyKF3X+34UV8ltDdAqp77zKj6nmduOHz+ul112mVapUkVXr16ta9as0bVr12qTJk20RYsWmpSUlKf+/Y0JccGJxR6cFMvsqrgken8DknAO516ZtHkOeE5V3xSRksAG3GusfqFmzZrUqFGDW2+9lRYtWlC2bFm2bdtGly5d+Oyzz9iyZQuzZ88mPDzcX1MwDMMo8vg7id5c4C1VffnUJHpAWVw5zzu941uBWM0iV1JBkWZsa9KkCRs3bmTTpk2MHj2aBx98kNTUVO677z7Cw8Pp0KEDkyZN8udUDMMwiiQBSaLnXbsNV72tDq5Gww6go6puy6K/AjO4ZWVuGzx4MPfccw9NmjTJVd+Fhe23WuzBhsVe/DSGOKBKhuOn8TQG7/jXISmCAAAgAElEQVR8XP3n9t7xK8CzvvSdV40hK3PbjBkzNDIyUgGdNm1anvouDGy/NTix2IOT4mpwKwcsERH1DGz3APeLSCvv+lbgCDBPRH4FPgLa+HE+WZrbmjdvzieffEKFChVy7sQwDKOY48+FoTxwA3AxcAnwb+BLYDKAqu4ENgJ3e+0740ORnvyQZm5buHAhnTp1Sje3rV27ls6dO3PgwAEeeughunXr5s9pGIZhFGn8KT6HAtOBD4EZuIVCgbIiUl6dyHwr8A7O6NYa6O9L/3k1uGVlbitfvjzXXHMN0dHRjB07lvPP920bzjAMozhSaOKzd+4RoKmqDshw7hxgvqo2z6G/AjO47dmzh/j4eC677DKGDRtGbGwsU6dO5c8//2TEiBF07do1130XBibEWezBhsVePMXnjBXcLsV5FNIruHntJuEMbr8BE/AWq+w++TG4LV26VLt27arPP/+8NmrU6CSDW4UKFUx8LqJY7MGJxV5wUEQMbuAquHXFCdGzgJ3eAgCAiFyEeztpA9AK+B6nR8T6YzI1atRg2LBhREREMGLECJYuXZpucDMMwzAc/hSfz8Y5nj8Fvgbm4TSHjEnyFFegR7yfZwG7/DWhNHPbwoULiYyM5NNPP2X//v3MmjWLOnXqmPhsGIZB4VRw+wboiROjN+AMbQNV9SMR+cC7Vga3SCxQ1R5Z9GcGN9tvDfQ0AoLFbrEXBEVCY1A9KVlephXccG8jxeDSY5TFJduLyqnfgja4JSQk6OWXX65hYWF6wQUXaGJiYp769ze23xqcWOzBSbE0uInIg7hynbuBq4FvReQocB3wmoj0Avri6j0vA34EfgE6+GtOWRncRo8eTefOnWnfvj3t2rVj9OjR/pqCYRhGkcff2VXjcUnylqrqXhH5G863cK+qzhaRh3D6QnNcvqRPcVqEX8iqetv06dMBSEhIYPXq1Rw+fNiqtxmGEbTk+olBRCqJSMsc2qRlV60GtFXVvd6lpUC416Y8UA/4DveksAxYrqp+WxjSDG6rVq1i9uzZVKhQgQceeIDDhw+zfft2jh07xu7duylZsqS/pmAYhlHk8Ul8FpFY4CrcE8ZK3PbQN6o6JJt74sjG4CYirXHpMdbgXlVdDgxS1aQs+suXwW3MmDEsXbqUihUr8tprrzFo0CAuv/xyvvnmG9auXUv79u157LHHKFOmDD179mTePL+tT3nGhDiLPdiw2Iuw+Ays8H4OAJ7xvq/K4Z4E4Hf+quB2HPcq6tne9fNxbyJtwC02u/BjdtVvvvlGly9frpGRkdq1a1cdN26cnn/++RobG6uNGzfWcePG6eOPP67bt2/XvIrb/saEuODEYg9OzgTxOVREauKE4/k+3pOWRO8enKM5CXhbVRO861txLugOqtoa6IMfs6tGRUVRqVIltm3bRkREBEOGDGHdunVERUVx1VVXER8fz8cff8y0adO4+uqr/TUNwzCMIo+v4vNI4HNgsar+KCINgPVZNT4lid5M4CbgEyB9W0lVd4pICtDQO+/37KrLli1j//79LFy4kNatW5OamsqTTz7J8OHD6dChA3/88QdffvklM2fO9Oc0DMMwijT+NLilAJHAUOBa4BjuCWGHevtcIrINqIh7M2kdcJn+JVSf2l+eDW5p5radO3cyYsQI3n77bQA2b97Mv/71L/bv38/FF1/MJ598wpw5c/ISbqFg+60We7BhsRdtjaExLq3Fr95xS+DxHO6JI5sKbt65Wt7PasDP+GBu0zxqDP3799fKlStrqVKl0s+tWLFC27dvr61atdJmzZppZGRkrvstTGy/NTix2IOTM0FjeAOXKTXZW0xW4fSD7Mi2gpuINAEWiMhK4AugKfCwj/PJNbfddhvTpk076dzgwYN56qmn+Omnn6hevTopKSn+Gt4wDOOMwVeNIVxV/yciGc/l9Fu0PNAFCMNlVB0E1Ma9otoeJz53UtWDIlIO9xbTx7mYe67497//zddff82xY8eoU6cOzzzzDDt37uT222+nXLlyNG3alDZt/FpZ1DAM44zA14Vhj4g0xL1eioj0AXZk1diXCm64HEqzvMWmAk57eCePceTIBx98QFxcHD169ODXX38F4KKLLqJbt24cOXKE5cuXs2TJEn8NbxiGccbgq8GtAe5/+hcBe3EFd25S1T+zuScOHyq4eeffAn5S1Vez6S/fBrfFixdz9OhRvvjiCwCeffZZ1q1bR1hYGIcPH6Z8+fJMmjQpV/0WJibEWezBhsVeRMVnXNqM67zvZYByvogX5GBw89oMAlbjtqWyFbMzfvJqcJs3b95J4nNoaKjGxMSoqur8+fM1JCQk1/0WJibEBScWe3BSpMVnVT0B3O99T1LVgz4uUNka3ESkOS7B3tPAIiBaRBr52HeuiYqKomLFiiedK1WqFEuXLgVgyZIllC9f3l/DG4ZhnDH4qjF86W0D/Qf3Cx4AVU3MrLEvBjcgApdU71rgfVwBn2uAF3IXgm/07dv3NPF50qRJ3H777elptj/55BN/DG0YhnFG4evCcLv3874M5xSXQfU0VHWgiAwAeuEMbmcD3YErROQGdftcvwLPe9cG42pCL/NlMkeSUzlneIxPE48b3R3IXHx+8MEH+eCDD7j22muZMWMG48ePp0ePTAvIGYZhBA3+Lu2ZLj6LyNPAIVUdm6HNHbjF5hAuHcYRVX0oi/7yJD5nzKo6atSodOfzJ598wquvvkrdunW58MILufvuu+nRowcxMb4tOIHAhDiLPdiw2AMjPvv0xCAit2R2XnN+vfRO794aOPE5RURuAAar6veqOkVEGuOeJq4FssxHoaqTcW9G0aRJE42OjvZl6pQoUYKyZctyyy230KFDB8qUKYOqsmbNGho3bsykSZOIjIzkl19+oWnTpvjabyCIjY0t0vPzJxZ7dKCnERAs9uiAjO3rVlK7DN/DcAnvfsJVY8uO24GuuPoNj+CeDL7A+RqaisiNuIyqPYHPgLYiUl5VD/gcQQ5ERUURFxfHli1buPDCC9mzZw/du3fnnnvu4emnn2bQoEGkpKQQFhbG5MmTC2pYwzCMMxafFgZVfSDjsYhUAN7N4bazgVLAAqAm7rXXEzjNIS1R3nNAWWA2rhTojcAVuIWjQKlbt266ttC6dWvKli3Lww8/TNmyZRk7dizt2rXLoQfDMIzgIE8ag4ichSvUE5FDuzg8nUFErgFG4RLmdVfVH0SkK/AULnVGOPA/YKKqjsukr1xnV21Ru0KmxrZrrrmGo0ePUqtWLY4dO8bhw4f5+OOPOSXlR5HD9lst9mDDYi+iBjdv4ZgHzPU+84GNwJgc7nkQpyvMwxncjuG2k6KAr4AmuMptO4AjOJPbMlx5zwIzuGVmbGvYsKEOHDgw/bhBgwYaHx/vc5+Bwsw+wYnFHpwE0uDmq8YwNsP3FOBPVd2awz33AvHAo7hMq728hehbL+9SgrrKbYhICLANl1gvywJAeSEqKorvv//+pHNNmzZlw4YNAPz+++8cP36cKlWqFOSwhmEYZyy+pt2+UlW/8T6LVXWriIzJqrFncGuA2zbqg9MUkr1rbYCSQIKIhIjI2TgxewdwLk6cLjD69u1L7969041tU6ZM4bzzzuOHH34gLCyMCy+8kFdffbXIbyMZhmEUFr4+MXQBhp1y7m+ZnAPSDW5X4BaeScAduCeIZNxrqderqnpaxXc4cXofcI2q5lgUwReDW3bGtl27dvH0008jIjzxxBPMmTPH6jwbhmF4ZCs+i8g9uF/oDYA/Mlwqh6v/3C+be+PIweDmnS8JbAeaqequbPrLs8HtVPF56tSpxMTEUKFCBZKTkzl27BgffvihT/0FEhPiLPZgw2IvguIzrk7COcAHQP0Mn8o5iRecnl01BZh3Spu3cE8KB30VRbQAxOchQ4boiy++qKqqL730kl5//fU+9xdITIgLTiz24KTIis+quh/YD/QFEJFqOINbWREpq6qbs7k9rYLbVm8xOe0VVGAqrqpbs+zmkR8yq9z21VdfER8fzzvvvMM555zD66+/7q/hDcMwzjh8TYnRE3gJqIV706g+rlxnpr/QM6ngdhdQFTghIluBSHXu5mXABWRTDS6/ZKYxbNmyhalTpwJQpUoVwsLC/DW8YRjGGYevFdx+Bi4DvlLV80TkUqCvqt6VzT1x+KYxnAPMV9XmOcwhVwa3FrUrpH/fuXNnevI8gMTERCpUqICI8NZbb5GQkMCwYZnq6EUK22+12IMNi70Iagz6lxawzPv5M1DC+/6/HO6JA6pkOH4aeCSTducAv/q696W50Bj69++vVatW1UaNGmmzZs1Ouvbiiy8qoMuXLz/tWlHF9luDE4s9OCnSFdw89olIWdyrpe+JyCs4MTkn7hSR30REcZXchojIEhFplaHN7cC5IvKriHwgIgW2r3Pbbbfx2WefnXZ+2bJlfPnll9SrV48FCxbQvHm2DyuGYRhBha8+hqtxaSsG46qxVQBG+nBfWnbVmsBVuApuq3Hps9uLSG2gP/CHqjYXkRm4cqBTcxFDlkRFRXHVVVexadMmgHTxeeTIkZQsWZIdO3bw/fffp28xGYZhGL5nV00SkfpAI1WdJiLhQEgOt2WVXTUJSLMZT8QJ2jU8UToBeDOn+eRkcEsztwFMmDDhJOF57ty59OrVi1deeYVzzjmH6dOnWzoMwzCMDPgqPt+JE34rq2pDEWkETFLVzjncF0cGAdo79wjQVFUHeMeDgH/inki+UNWbsugr1wa3U81tR48e5aGHHuLFF18kJiaGSZMm8c4771C3bt0c+yoKmBBnsQcbFnvRFp9X4vIbrchw7hcf7ovjZAH6UtxrrsO9n0nAQeAXXA1oBe7OqV9fxedTzW2rVq3SqlWrap06dTQsLEwBrV27tu7YsSMXEk7gMCEuOLHYg5MzQXw+pqrH0w5EJNT7Je4zItISt010NXAbcKX3c4aqtgBGeItFqyy6yDVRUVFUrFgx/bhFixbEx8fTvn17li5dSkhICF9//TU1atQoqCENwzDOeHwVn78RkX8ApUWkCy5/0jxfBxGResAnwM3AEFzupbm4ugwdPM2iLy49xm++Tz97+vbte5rruWrVqtSuXZtWrQps/TEMwyhW+KoxlMBlSO2KE44/B97UHG5O0xiA0bisqn96lyKA2uoquz2DexOpIS6v0i2qeiyTvnw2uGVlbsuoMZQtW5YbbriB119/nQoVKmTZV1HC9lst9mDDYi+CGgNQz9c9qdx8OF17uJ5TEuxl98lJY0gztjVr1kw3bdqkzZo10xkzZmiDBg0U0Jo1a2r9+vU1JCRE69ataxrDGYDFHpxY7AUHBagxzE77IiIf53aFEpEHPYPbxyLyg4gc895KSrseJiL/w/kazvOeHvJNZsa25s2bs2DBAi655BLmzp1LXFwcderU4aeffjKNwTAMIwM5LQwZy5o1yEP/9+JE5ntwNaDHnnL9GK7kZyouId8VItIhD+OcRFRUFJUrV2bLli1ceOGFrFu3ji5dupxW4tMwDMM4nZzEZ83ie45kKO85F3hLVV8Wke4Z26iqikhXXDnPZOAsX8bJyuCW0dgGULdu3XRjWxrvvvvuX+3j4nwLxjAMI4jIqYJbKn85lUsDh9Mu4X6vl8+2cx8yrIpICLAcV+95oqpmmuY0Nwa3zKq2xcbGMnXqVP78809GjBhB165ds5t6kcSEOIs92LDYi6D4nJ8PbuvoOO4JYJX32QKMy9BmEM7YthrnY1gENM+p75zE58yqtq1Zs0bXrl2rFSpU0GnTpuVGsykymBAXnFjswUlRFp/zw724oj5XApeoakvgG6APgIg0B+7EFeppBXT2Fo8r8jvwqcY2gIiICJo0aZLfrg3DMIo9flkYMugL1YC2qrrXu7QVSPuNHYGXakNVU4DFuAysa/M7ft++fendu3e6sW3KlCnMmjWLOnXqcODAAR566CG6deuW32EMwzCKJT4Z3PLUsdMXSgBtcCL3MqAKTp/YjXsb6SPggHeuIbBKVTN9K8kXg1t2VdvSGDx4MPfcc88Z+fRg+60We7BhsRc/jSGOzBPonZ3h3B3AT8C3wCTgZV/69sXgVrly5ZM0hoSEBL388ss1LCxML7jgAk1MTMzN9lyRwPZbgxOLPTgprhoD/FXB7QCQ5jj7XEQ6et+/wonT5XEpM6oXxKC33XYb06ZNO+nc6NGj6dy5M+3bt6ddu3aMHj26IIYyDMModviaRC+v3I7LoPoOcLOqLvGyrM4AmuLKg16EWxC+BDqKSC1V3Z6fQf/973+fljxv+vTpACQkJLB69WoOHz7MmDFj8jOMYRhGscSfC0NaBbd5QBjwmojgnUsTNj702iXjajS84kvHORncPvjgA+Li4k6q3Pbwww+zb9++9LaVKlXKS0yGYRjFHr+Jz3CywU1ErgFG4d5U6q6qP3ht6gIxOIPbo6o6MYu+8mVw69GjB02bNmXnzp3UqFGDtWvXMn/+/AKLtTAwIc5iDzYs9mImPnsLTgLwO/Ced9wOlxdp1SntygM7gF1A9Zz6zYvBrVKlSjpixAhVVR0xYoRWqlTJZ9GmqGBCXHBisQcnxVl8Lg/coKo3eakvxuBqOVQXkSoZ2j2Lcz0nAp3yO2hmBjfDMAzDN/ymMXgmt1BguojMwnkXPga6eecTRKQOUA8nPn8L/A1Yl9+xM6vclpKSwn//+18aNWpEvXr1SE1Nze8whmEYxRJ/awwpQCRwKzAYWA/UAiao6nNeZtVPgM1AOWCDql6aRV/5Mrj16NHjJE2hZ8+ezJvnc3XSIoHtt1rswYbFXjw1hjic23km0ME7NxXo432/Hxjqfb8NeNWXfvNicGvYsKF26tRJzz333PSfZxq23xqcWOzBSXHWGMoBS3BbRB+KyHbc08MUEekFXAg8IyIncFXcbhGRfDvPMjO4Va5cmfDwcNavX094eDiVK1fO7zCGYRjFEn8b3MoDXVT1J098/hI4AXyoqrOB2SLSGQjHCdDfq+rw/A6amcEtMTGRUqVK0ahRI2rUqEFCQkJ+hzEMwyiWFJb4/AbO1PYx8FDGdqr6tYhE56bvvBrcNmzYkN7WDG6GYRiZ47eFQVUHisgVQBTO7fw+cBnOy7A0k1u2qur9WfV3isGNGVeUOa1NbGwsH330ETExMSQnJ3PkyBFiY2MBSElJSf+e2fGZwKFDh864ORcUFntsoKcRECz22MAM7qsYkZcPfxnc/gQ64BaFE8DYDG3GAJtw6bev96XfrMTnX375RZs1a6ZJSUm6fv16LVOmjP7++++qqtq4cWPdvn27qqpu375dcxKwiyImxAUnFntwUpzF5/LADd5i8CHOq3ACl3W1l4h0x9VrGIAr1POoiGRbRzo7fvvtNzp06EB4eDihoaGUKVOGWbNmAXDVVVelC9LTpk3j6quvzk9chmEYxZZC0RiACTiNIRmnMcxW1dki8iiu3Geq9/kZV9oz60RI2dC8eXMee+wxevfuzZIlS9i9ezcjR47k7LPPZvjw4Vx33XVMmTKFevXqMXPmzAKI0jAMo/hRWAa3JP7SGNbhFoZHPIPbTOA4UBY4C3hLVe/KpK9sDW5p5raYmBjmzJlD6dKlqV+/PqVKleK+++7zV4iFipl9LPZgw2IPjMHN36+rbsXlP/o3MExVU0XkezzxWVW/EJEXgL/jUmbEA6sz60hVJ+O8DjRp0kQfuCnzraAVK1Zw1llnkZqayoYNG7j77ruJjo4u4LACQ2xsbLGJJbdY7NGBnkZAsNijAzK2vxeGNINbbaCniJTCPR1cKSIp6rwMy4FbgP8DqgLv5XWwbdu28fLLL7Nu3Tp2795N8+bNCQ31d4iGYRjFi0IxuOGeHOoDvYCuwBhPYzgL9zTRGaiIE6C35GfAXbt20bp1a0qVKkVkZCSNGjXKVwCGYRjBRmGJz2+o6sveW0gZuQj3lDAf97rqG0AP4Jfs+s7M4BY3uju1a9fmhRde4LHHHqN06dJ07dqVrl27FlBEhmEYwUGhVXDzjp8GDqnqWO+4D3CFqg7wjm8G2msmRrecKrht3ryZp556Kr1C265du6hRowZ9+/alS5cufouxMDEhzmIPNiz24ik+lwOWiMiPqnoTLuX2ABGJU9WPAAEaich6r/1CnAZxGqeKz5mJMqVLl+azzz5j8uTJ1K5dm/vvv58NGzYUG/HKhLjoQE8jIFjs0YGeRkAozuLzqUn0unByIZ79ODd0HZzP4Q/gX3kdrF69eixdupSYmBgaNGjAzz//zPnn+5Z+3DAMw3AUdhK9NUC1DM2qAilABWCb1+ZQXsds3749ffr04ZZbbiE8PJyGDRty112nWSIMwzCMbCgsg1s48F/gGBAGHMS9nnoX0BKXRykE2Ah8nqZBnNJXlga3jJXbkpOT6dOnD2+//Xaxq7lg+60We7BhsQdnBbdHgccztH8CeDinfrNKgLd371698MILNTw8XJs2bapLlizxPcPUGYAlFAtOLPbgJJBJ9ArL4NYQuFpEwD0ZXO09TWwFeojIF0BdoAbwZF4HGzRoECkpKbz66qvcdNNNHD58ON8BGIZhBBuFZXD7HUhSVRWRObhXWGeLSGXgLeBGIBZYAXySl4EOHDhAbGwshw4donfv3pQsWZKSJUsWTBSGYRhBhN/Sbp9icLvTe5TBO5f2vQbO6fwi8CPwjKpuy8t4GzdupHr16vTs2ZPo6GgGDBhAUlJS/oIwDMMIQgrN4CYi1wCjcG8ldVfVH0SkF64Ww3GcGP0VMFxVUzPpK1uD27p167j33ntp2bIlCQkJ7N27l44dOzJs2DC/xVfYmBBnsQcbFnvxFJ/TKrh9DPyAeytpIvCVd70PcMRr8yvu6eHunPrNTHzesWOHlilTRt944w1VVf3666+1S5cu+ZBqih4mxAUnFntwUpzF58yS6O3FZVet4i0UAK1U9YiI/Be4Dng9twOFh4eTkpJCx44dAfj2229p1apVAYRgGIYRXBRWEr1ZwONAd1wK7pK4p4mVuLeU6ojIJlzKjNl5GW/jxo2ce+65dOzYkaSkJKpVq8bixYsLIBLDMIzgwm/is6oOxJXr7IXLnPorMBC4Brjee7rZArwJrAWO4rSGh3PqOy27asYMqykpKaxdu5aYmBiOHDlCr169eP31XD94GIZhBD2Bzq5aCac/XA/swxnhPlLV6Zn0la34nJiYyD333EPTpk3ZtGkTx44do2rVqrz66qt+i6+wMSHOYg82LPbiLT7vBVYBO3ECc0fv+t+Bb4D13mcy8FpO/WblfK5WrZo+++yzqqr6+OOP6/333583laaIYkJccGKxByfBID7/DiQBT+Hc0G8CTXFPCRfiXlU9DGwGns3LQAcOHCAkJITZs2czY8YMGjRowNtvv10AIRiGYQQXhSU+f4jbBiqPq8EQJiLlcXmUVgFf47Ks7vA+uWbjxo3UqVOHyMhIfv75Z6pUqWLOZ8MwjDwQaIPbI0CYqj7ntX8COKI5ZFc1g1vwYbFb7MFGcdYY4oAqp5yLwhXr+Q33ptJa4GdgNS5PUp6yq5rBrXhjsQcnFnvBQS40Br+9rprNQvQtTlO4yVsYjqpqKyAauBLYlZd+zeBmGIZRMPhbfAZARM4F/lBVFZGPgLOAaTjj25Xea6u1cGa3T/MyhhncDMMwCobCemK4FvhVRFbinM87gUtxWVW3AbtxKbdfVdWEnDozg5thGIb/8Kv4nOWgniiN2z66GBiCK+bzJS5v0oFM7slWfN69ezc33ngj559/PqNGjWLVqlW8//77jB492q+xFCYmxFnswYbFHhjx2a9bSSLyIHAPzrPwi3f6EG4rCeAOXHK9lRnm0hT436l9qepknAGOJk2aaHR09EnXX3rpJSpVqkSpUqWIjo4mNjaWTp06cWq7M5nY2NhiFU9usNijAz2NgGCxRwdkbH9vJd2LE5QvBi5R1ZY4A1sV73ocMNsTn7vitpkq5XaQrVu3EhMTw3PPPcfKlStp2bIlK1eu5B//+EeBBGEYhhFM+Nvg1gCYC7ylqku8S0txIjO4RWKqiPyCW6S24dzQuWLw4MG88MILHDx4kI4dOzJ//vz8B2AYhhGkFGoSPe/cI0BTVR3gHYcAy4FzgYmqmqkjLaPGUKVK1bZPjn+DFrUr8O233/Lyyy9TuXJlkpKSCA0NZfr003LwFQtsv9ViDzYs9uJpcDs1id4GXMW2KzO0+Qz3lPAZsAhonlO/GQ1uw4YN01q1amn9+vW1evXqWqJECe3WrVv+nCBFFDP7BCcWe3BSnA1u5YEbgLpAP1yepOuAlzK0eRG4GZcrKRa4IjcDjB49mm3bthEXF8fUqVMpW7YszzzzTEHM3TAMIygprCR6M3FO55sB9T6ISFXcNlJrnMZwOTAmt2OlpqbStm1b1q1bR61atWjfvn3BBGEYhhGEFFYFt9pADWAh8F2GZjVx20dTgI7Al6qao3J8JDn1pOOQkBBWrlzJjh07qFevHr/++mvBBGEYhhGEBEJ8jgKeVNXLM5yLBh5R1R7Z9HWawS0+Pp5Ro0aRmJiIiNCjRw+SkpIICwvj+uuv91dYAcOEOIs92LDYg0N8XgksA7bjZV3FPVVsAPYDc33pN0183r59u3711Ve6d+9ePXDggDZs2FDbtGmj8+bNK1DRpqhgQlxwYrEHJ8FQwS0F+EVVVUSux+kOaTmRjgADcE8MV+Wm85o1a7J7924uvfRSUlNTiY+P5+KLL6ZHjywfPAzDMIwcKCzx+Q+ggYgk48xt27wVDKA0TpwuKyJbgTtU9XNfx2nZsiUrVqwgLi6OqKgo/vWvfxVsIIZhGEFGQCu4eW1ScFtMKcBoVZ2dRV8nGdxmzvwrid6RI0cYNGgQ/fr1Iyoqym/xBBrbb7XYgw2LvXhqDHFkXsHtqwzHtbyfDbz2DXPqN01j2Lx5s0ZFRWmZMmW0evXqOn78+ALZi3mivy0AAAolSURBVCuq2H5rcGKxByfF2eBWDlgiIntFZJVXj+ElIFJEqohIa+BjEVkNzPYWhvN87TwkJIRy5coxYMAA1q9fz8SJE1mzZo0/4jAMwwgaCsv5fAmuzkJrnNO5Kk58DsVpCs2AvjgvwxZfO9+4cSMxMTEsXLiQTp06sWvXLmbNmlXgQRiGYQQTARWfRaQk8LaInMAtUjuAxJz6TjO4dezYMW07Kl18fuCBB/wRjmEYRtAQcPE5Q9sLcHWgm6nqiUz6ytLgtmfPHuLj47nssssYNizT5KzFAhPiLPZgw2IvnuJzmsHtPe+4Hc7QtipDm1TgV+Ao8J0v/WY0uC1dulS7du2qzz//vDZq1EhXr15dUFpNkcOEuODEYg9OirP4XB64QVVv8uoujAE+B6qLSFoVtyPAceBmVe2Um85r1KjBxIkTiYiIYMSIEURERLBt27YCDcAwDCPYKBSNQURmAbuBj4Fu3vkET2MIA95R1Zm5HWPx4sW8++67tGjRgsjISDZs2MDNN99cgFEYhmEEH/7WGFKASOBWYDCwHqgFTFDV50SkH/Au7qlBgXjgGlVdmUlfZnCz/dZATyMgWOwWe0FQlDSGOKAKLuVFB+/cVKBPhjZ5Nrj1799fq1SpomXKlNFx48YV4G5c0cT2W4MTiz04Kc5J9MoBS4BGwNUiAu511atEJEVVZ6vqdk9/+Ni7dh7u9dYcufXWW4mPj+eHH35gyJAh/onAMAwjyCgsg9vFQHVVLYkr1pOgqrNFpJKIlAIGARuByoDP1uWQkBBiYmJISkqidevWtG7dmgULFvghDMMwjOChsAxub6jqEu/SbqCN9z0CV72tjnd+g6r6vDB07NiRTZs20aNHD1auPE2WMAzDMPJAICq4PQI0VdUB3vFHOONbObKp4paZwW3MmDEsXryYo0eP8sUXX/gtjqKCCXEWe7BhsRdP8flUg9tA3NtHt3nHDwA7gdU4XWG5L/2mic/ffPONzps3T0uVKlVQ+kyRxoS44MRiD06CxeDWGpdZ9RvgkHe9mfezDO6JoY2I/MfXzqOioqhYsWJBztcwDCPo8dvCcIrB7RngS+A13CupAKjqQFWtoarnANcBB4HHfR2jb9++9O7dm2PH/r+9+w+9q67jOP585XJq1mw/jNHX3MwFNljLrPbFtdZIWStiooXfBBUG/SQMovBLEfOP/gghJYhImwsjZlBmY/2xyTQG4Y/pNudX1jcn3y8kc32V3MwaovLuj/M+271f7rZv27333HvP6wGHc8/nfPju8z473M89n8857/MGQ0NDbNq0qZ0hmJnVUrcecNsIfIliyOgDwKsR8cFpdb9OkZJ7TswwiR7A4cOHGR0dZfPmzR2Lo1d4vNWx141jH8w5hklO84Bbli0Exss6p1vKOYaIiImJiVi6dGlbxuB6ncdb68mx19MgP+BWugp4IB9wmw+sKx9wk/Qe4M/ADyPi8S61x8zMTqLTk88ARMTiiFgUxVzC74FvZqdwLvBHzjCJ3sjICMPDw4yPj3uOwcysTbp1xXAyXwZWAfMk3Zplt0aLJHqtbNmypVPtMjOrrY5OPneKpH9TzEnUzXzgldPWGkyOvZ4ce/tcGhELZlKx6iuGMzUeM51dHyCSnqpj3ODYHXv9VBl7V+YYzMysf7hjMDOzJv3aMdxTdQMqUte4wbHXlWOvQF9OPpuZWef06xWDmZl1iDsGMzNr0lcdg6S1ksYlHZR0e9XtaTdJ90makjTWUDZX0sOSns/1e7Nckn6Wx2K/pCtP/pd7n6RLJD0q6YCk5yTdluUDH7+k8yQ9KemZjP2OLF8s6YmM/XeZKQBJs3P7YO5fVGX7z5akcyTtlbQtt+sS96SkZyXtk/RUlvXE+d43HYOkc4CfA5+jyNg6IunD1baq7X4NrJ1WdjuwMyKWADtzG4rjsCSXrwK/6FIbO+Ut4LsRcQWwAvhW/v/WIf43gDUR8RFgObBW0grgJ8BdGfurwIasv4EiQ/HlwF1Zr5/dBhxo2K5L3ACfiYjlDc8r9Mb5PtNse1UvwDCwvWF7FBitul0diHMRMNawPQ4sjIYstPn5l8BIq3qDsAB/Aq6pW/zABcAe4JMUT73OyvLj5z+wHRjOz7Oynqpu+xnGO0TxBbgG2AaoDnFnDJPA/GllPXG+980VA/B+4B8N2y9m2aB7X0S8BJDri7N8YI9HDhF8FHiCmsSfwyn7gCmKl1q9AByJiLeySmN8x2PP/UeBed1tcdvcDXwfKN/BMo96xA3Fa453SHo63zcDPXK+91NKDLUoq/O9tgN5PCRdCPwB+E5EvJap2ltWbVHWt/FHxNvAckkXUWQcvqJVtVwPROySvgBMRcTTklaXxS2qDlTcDa6OiEOSLgYelvS3U9Ttauz9dMXwInBJw/YQcKiitnTTPyUtBMj1VJYP3PGQ9E6KTuG3EfFgFtcmfoCIOAL8hWKe5SJJ5Y+3xviOx5775wD/6m5L2+Jq4IuSJoEHKIaT7mbw4wYgIg7leorix8An6JHzvZ86ht3Akrxj4VzgRmBrxW3qhq3ALfn5Foqx97L85rxbYQVwtLwE7UcqLg02AQci4qcNuwY+fkkL8koBSecDn6WYjH0UuCGrTY+9PCY3AI9EDjz3k4gYjYihKN7TciNFHDcx4HEDSHqXpHeXn4FrgTF65XyvegLm/5ysWQf8nWL89QdVt6cD8W0BXgLepPiFsIFiDHUn8Hyu52ZdUdyl9QLwLHBV1e0/y9hXUlwa7wf25bKuDvEDy4C9GfsY8KMsvwx4EjhI8Xrc2Vl+Xm4fzP2XVR1DG47BamBbXeLOGJ/J5bny+6xXznenxDAzsyb9NJRkZmZd4I7BzMyauGMwM7Mm7hjMzKyJOwYzM2vST08+m3WUpLcpbgUsrY+IyYqaY1YZ365qliS9HhEXdvHfmxUncgKZ9QwPJZnNkKSFknZl/vwxSZ/K8rWS9uT7FHZm2VxJD2Xu/MclLcvyjZLukbQDuD+T590paXfW/VqFIZoBHkoya3R+ZjgFmIiI66bt/wpFCugf5/tBLpC0ALgXWBURE5LmZt07gL0RsV7SGuB+inctAHwMWBkRxzKr5tGI+Lik2cBfJe2IiIlOBmp2Ku4YzE44FhHLT7F/N3BfJvt7KCL2ZVbQXeUXeUSUSd1WAtdn2SOS5kmak/u2RsSx/HwtsExSmRtoDsXLWNwxWGXcMZjNUETskrQK+DzwG0l3Akdonf74VGmS/zOt3rcjYntbG2t2FjzHYDZDki6leH/AvRSZYK8EHgM+LWlx1imHknYBN2XZauCViHitxZ/dDnwjr0KQ9KHMtmlWGV8xmM3cauB7kt4EXgdujoiXc57gQUnvoMiffw2wEdgsaT/wX06kUp7uVxSvc92TqcdfBtZ3Mgiz0/HtqmZm1sRDSWZm1sQdg5mZNXHHYGZmTdwxmJlZE3cMZmbWxB2DmZk1ccdgZmZN/gcWc848bGMvLQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from xgboost import plot_importance\n",
    "plot_importance(xgb7_1)\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAFlNJREFUeJzt3X+w3XWd3/HnyyDR6Vblx7XDJrCJY7Zr1G0cLpEZK7OLgqFawkxhDcMKduik6yztdqxbY1uxzeoMtDOldcpYsoLiT2SxrndKmCwrsO20yubyYwmBUi4xhWuY5WrQpXWFRt7943ziHg833O+5Obk3uXk+Zs7c7/fz/Xw/388HTs7rfH+c7zdVhSRJr1jsDkiSjg4GgiQJMBAkSY2BIEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNScsdgeGceqpp9aqVasWuxuSdEy57777vl9VY3PVO6YCYdWqVUxOTi52NyTpmJLkf3ep5yEjSRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEnCM/VL5aLBqy+0vKdt7zXsXoSeSNFruIUiSAANBktQYCJIkwECQJDUGgiQJMBAkSY2BIEkCOgZCkg1JHksylWTLLMs/nOSRJA8l+VaSX+pbdkWSx9vrir7yM5Psam1+OklGMyRJ0nzMGQhJlgHXAxcAa4FLk6wdqPYAMF5VvwrcBvybtu7JwCeAtwPrgU8kOamt8xlgM7CmvTYc9mgkSfPWZQ9hPTBVVXuq6gXgFmBjf4WquruqftxmvwOsbNPvAe6sqv1V9SxwJ7AhyWnAa6rq21VVwBeAi0YwHknSPHUJhBXAU33z063sUK4E7phj3RVtes42k2xOMplkcmZmpkN3JUnz0SUQZju2X7NWTH4TGAf+7Rzrdm6zqrZV1XhVjY+NjXXoriRpProEwjRwet/8SmDfYKUk7wb+BXBhVT0/x7rT/NVhpUO2KUlaOF0CYSewJsnqJCcCm4CJ/gpJ3gbcQC8MnulbtAM4P8lJ7WTy+cCOqnoaeC7J2e3qosuBb45gPJKkeZrz9tdVdSDJVfQ+3JcBN1XV7iRbgcmqmqB3iOgXgD9oV48+WVUXVtX+JL9HL1QAtlbV/jb9IeDzwKvpnXO4A0nSoun0PISq2g5sHyi7um/63S+z7k3ATbOUTwJv6dxTSdIR5S+VJUmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAR0DIcmGJI8lmUqyZZbl5yS5P8mBJBf3lf96kgf7Xj9JclFb9vkk3+1btm50w5IkDWvOB+QkWQZcD5xH71nIO5NMVNUjfdWeBD4IfKR/3aq6G1jX2jkZmAL+qK/K71bVbYczAEnSaHR5Ytp6YKqq9gAkuQXYCPwsEKpqb1v24su0czFwR1X9eN69lSQdMV0OGa0Anuqbn25lw9oEfHWg7FNJHkpyXZLl82hTkjQiXQIhs5TVMBtJchrwVmBHX/HHgF8BzgJOBj56iHU3J5lMMjkzMzPMZiVJQ+gSCNPA6X3zK4F9Q27nN4BvVNX/O1hQVU9Xz/PA5+gdmnqJqtpWVeNVNT42NjbkZiVJXXUJhJ3AmiSrk5xI79DPxJDbuZSBw0Vtr4EkAS4CHh6yTUnSCM0ZCFV1ALiK3uGeR4Fbq2p3kq1JLgRIclaSaeAS4IYkuw+un2QVvT2MPxlo+stJdgG7gFOBTx7+cCRJ89XlKiOqajuwfaDs6r7pnfQOJc227l5mOQldVecO01FJ0pHlL5UlSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKkxECRJgIEgSWoMBEkSYCBIkhoDQZIEGAiSpMZAkCQBBoIkqekUCEk2JHksyVSSLbMsPyfJ/UkOJLl4YNlPkzzYXhN95auT3Jvk8SRfa4/nlCQtkjkDIcky4HrgAmAtcGmStQPVngQ+CHxllib+sqrWtdeFfeXXAtdV1RrgWeDKefRfkjQiXfYQ1gNTVbWnql4AbgE29leoqr1V9RDwYpeNJglwLnBbK7oZuKhzryVJI9clEFYAT/XNTzPLM5JfxquSTCb5TpKDH/qnAD+sqgNztZlkc1t/cmZmZojNSpKGcUKHOpmlrIbYxhlVtS/JG4C7kuwC/qJrm1W1DdgGMD4+Psx2JUlD6LKHMA2c3je/EtjXdQNVta/93QPcA7wN+D7wuiQHA2moNiVJo9clEHYCa9pVQScCm4CJOdYBIMlJSZa36VOBdwCPVFUBdwMHr0i6AvjmsJ2XJI3OnIHQjvNfBewAHgVurardSbYmuRAgyVlJpoFLgBuS7G6rvwmYTPJn9ALgmqp6pC37KPDhJFP0zincOMqBSZKG0+UcAlW1Hdg+UHZ13/ROeod9Btf7H8BbD9HmHnpXMEmSjgL+UlmSBBgIkqTGQJAkAQaCJKkxECRJgIEgSWoMBEkSYCBIkhoDQZIEGAiSpMZAkCQBBoIkqTEQJEmAgSBJagwESRLQMRCSbEjyWJKpJFtmWX5OkvuTHEhycV/5uiTfTrI7yUNJ3t+37PNJvpvkwfZaN5ohSZLmY84H5CRZBlwPnEfv+co7k0z0PfkM4Engg8BHBlb/MXB5VT2e5BeB+5LsqKoftuW/W1W3He4gJEmHr8sT09YDU+0JZyS5BdgI/CwQqmpvW/Zi/4pV9b/6pvcleQYYA36IJOmo0uWQ0Qrgqb756VY2lCTrgROBJ/qKP9UOJV2XZPmwbUqSRqdLIGSWshpmI0lOA74I/P2qOrgX8THgV4CzgJOBjx5i3c1JJpNMzszMDLNZSdIQuhwymgZO75tfCezruoEkrwFuB/5lVX3nYHlVPd0mn0/yOV56/uFgvW3ANoDx8fGhgqjfqi23v6Rs7zXvnW9zh+1o648kddlD2AmsSbI6yYnAJmCiS+Ot/jeAL1TVHwwsO639DXAR8PAwHZckjdacgVBVB4CrgB3Ao8CtVbU7ydYkFwIkOSvJNHAJcEOS3W313wDOAT44y+WlX06yC9gFnAp8cqQjkyQNpcshI6pqO7B9oOzqvumd9A4lDa73JeBLh2jz3KF6Kkk6ovylsiQJMBAkSY2BIEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAkwECRJjYEgSQIMBElSYyBIkgADQZLUGAiSJKBjICTZkOSxJFNJtsyy/Jwk9yc5kOTigWVXJHm8va7oKz8zya7W5qfbk9MkSYtkzkBIsgy4HrgAWAtcmmTtQLUngQ8CXxlY92TgE8DbgfXAJ5Kc1BZ/BtgMrGmvDfMehSTpsHXZQ1gPTFXVnqp6AbgF2Nhfoar2VtVDwIsD674HuLOq9lfVs8CdwIb2POXXVNW3q6qAL9B7rrIkaZF0CYQVwFN989OtrItDrbuiTc+nTUnSEdAlEGY7tl8d2z/Uup3bTLI5yWSSyZmZmY6blSQNq0sgTAOn982vBPZ1bP9Q60636TnbrKptVTVeVeNjY2MdNytJGlaXQNgJrEmyOsmJwCZgomP7O4Dzk5zUTiafD+yoqqeB55Kc3a4uuhz45jz6L0kakTkDoaoOAFfR+3B/FLi1qnYn2ZrkQoAkZyWZBi4Bbkiyu627H/g9eqGyE9jaygA+BHwWmAKeAO4Y6cgkSUM5oUulqtoObB8ou7pveic/fwiov95NwE2zlE8Cbxmms5KkI8dfKkuSAANBktQYCJIkwECQJDUGgiQJMBAkSU2ny06XslVbbn9J2d5r3nvMtC9Jo+IegiQJMBAkSY2BIEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAnoGAhJNiR5LMlUki2zLF+e5Gtt+b1JVrXyy5I82Pd6Mcm6tuye1ubBZa8f5cAkScOZMxCSLAOuBy4A1gKXJlk7UO1K4NmqeiNwHXAtQFV9uarWVdU64APA3qp6sG+9yw4ur6pnRjAeSdI8ddlDWA9MVdWeqnoBuAXYOFBnI3Bzm74NeFeSDNS5FPjq4XRWknTkdAmEFcBTffPTrWzWOlV1APgRcMpAnffz0kD4XDtc9PFZAkSStIC6BMJsH9Q1TJ0kbwd+XFUP9y2/rKreCryzvT4w68aTzUkmk0zOzMx06K4kaT66BMI0cHrf/Epg36HqJDkBeC2wv2/5Jgb2Dqrqe+3vc8BX6B2aeomq2lZV41U1PjY21qG7kqT56BIIO4E1SVYnOZHeh/vEQJ0J4Io2fTFwV1UVQJJXAJfQO/dAKzshyalt+pXA+4CHkSQtmjkfkFNVB5JcBewAlgE3VdXuJFuByaqaAG4Evphkit6ewaa+Js4BpqtqT1/ZcmBHC4NlwB8Dvz+SEUmS5qXTE9OqajuwfaDs6r7pn9DbC5ht3XuAswfK/i9w5pB9lSQdQf5SWZIEGAiSpMZAkCQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqOt3+WnNbteX2l5Ttvea9i9ATSZqfTnsISTYkeSzJVJItsyxfnuRrbfm9SVa18lVJ/jLJg+31n/rWOTPJrrbOp5PM9lxmSdICmTMQkiwDrgcuANYClyZZO1DtSuDZqnojcB1wbd+yJ6pqXXv9Vl/5Z4DNwJr22jD/YUiSDleXPYT1wFRV7amqF+g9G3njQJ2NwM1t+jbgXS/3jT/JacBrqurb7dnLXwAuGrr3kqSR6RIIK4Cn+uanW9msdarqAPAj4JS2bHWSB5L8SZJ39tWfnqNNSdIC6nJSebZv+tWxztPAGVX1gyRnAn+Y5M0d2+w1nGymd2iJM844o0N3JUnz0WUPYRo4vW9+JbDvUHWSnAC8FthfVc9X1Q8Aquo+4Angl1v9lXO0SVtvW1WNV9X42NhYh+5KkuajSyDsBNYkWZ3kRGATMDFQZwK4ok1fDNxVVZVkrJ2UJskb6J083lNVTwPPJTm7nWu4HPjmCMYjSZqnOQ8ZVdWBJFcBO4BlwE1VtTvJVmCyqiaAG4EvJpkC9tMLDYBzgK1JDgA/BX6rqva3ZR8CPg+8GrijvSRJi6TTD9OqajuwfaDs6r7pnwCXzLLe14GvH6LNSeAtw3RWknTkeOsKSRJgIEiSGgNBkgQYCJKkxrudHoJ3L5V0vHEPQZIEGAiSpMZAkCQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEdAyHJhiSPJZlKsmWW5cuTfK0tvzfJqlZ+XpL7kuxqf8/tW+ee1uaD7fX6UQ1KkjS8OW9u156JfD1wHjAN7EwyUVWP9FW7Eni2qt6YZBNwLfB+4PvA362qfUneQu8xnCv61rusPTlNkrTIuuwhrAemqmpPVb0A3AJsHKizEbi5Td8GvCtJquqBqtrXyncDr0qyfBQdlySNVpdAWAE81Tc/zc9/y/+5OlV1APgRcMpAnb8HPFBVz/eVfa4dLvp4ksy28SSbk0wmmZyZmenQXUnSfHQJhNk+qGuYOkneTO8w0j/sW35ZVb0VeGd7fWC2jVfVtqoar6rxsbGxDt2VJM1Hl0CYBk7vm18J7DtUnSQnAK8F9rf5lcA3gMur6omDK1TV99rf54Cv0Ds0JUlaJF0CYSewJsnqJCcCm4CJgToTwBVt+mLgrqqqJK8Dbgc+VlX//WDlJCckObVNvxJ4H/Dw4Q1FknQ45gyEdk7gKnpXCD0K3FpVu5NsTXJhq3YjcEqSKeDDwMFLU68C3gh8fODy0uXAjiQPAQ8C3wN+f5QDkyQNp9MzlatqO7B9oOzqvumfAJfMst4ngU8eotkzu3dTko4fg890X6jnuftLZUkS0HEPQRo0+A0GFu5bjKQjw0CQpCEt1iGdI81DRpIkwECQJDUGgiQJ8ByClghPch9b/P91dDIQJGlEjvWTzQaCAL+xSTIQjnl+kEvdHevf4I80A0ELYrGC63jbro5Ox0oQGQgaqSP9QXisf9Ae6/3X0mYgHGeOtg+ko60/Wly+HxaXgXCU8R/E8cn/7zoaGAh6WX5QLS7/+2shdQqEJBuA/wAsAz5bVdcMLF8OfIHeMw5+ALy/qva2ZR8DrgR+CvzjqtrRpU0dHj9I1MWx/j4Z1cnaxTrpe7SdbJ4zEJIsA64HzqP37OSdSSaq6pG+alcCz1bVG5NsAq4F3p9kLb1Hbr4Z+EXgj5P8cltnrjbV51j/hzus4228wzra/vss9MUEvheOjC57COuBqaraA5DkFmAj0P/hvRH4V236NuA/Jkkrv6Wqnge+2x6xub7Vm6tNacEdbR+00kLqEggrgKf65qeBtx+qTlUdSPIj4JRW/p2BdVe06bnalI4ax0pQHKqfo+r/sfrf4Vg7lLRYUlUvXyG5BHhPVf2DNv8BYH1V/aO+Ortbnek2/wS9PYGtwLer6kut/EZ6z2Z+xVxt9rW9GdjcZv8m8Nj8hwvAqcD3D7ONY4njXdoc79I2qvH+UlWNzVWpyx7CNHB63/xKYN8h6kwnOQF4LbB/jnXnahOAqtoGbOvQz06STFbV+KjaO9o53qXN8S5tCz3eLs9D2AmsSbI6yYn0ThJPDNSZAK5o0xcDd1Vv12MC2JRkeZLVwBrgTzu2KUlaQHPuIbRzAlcBO+hdInpTVe1OshWYrKoJ4Ebgi+2k8X56H/C0erfSO1l8APjtqvopwGxtjn54kqSu5jyHsNQk2dwOQx0XHO/S5niXtoUe73EXCJKk2flMZUkScJwFQpINSR5LMpVky2L3Z9SS3JTkmSQP95WdnOTOJI+3vyctZh9HKcnpSe5O8miS3Ul+p5UvyTEneVWSP03yZ228/7qVr05ybxvv19qFGktCkmVJHkjyX9r8kh0rQJK9SXYleTDJZCtbsPfzcRMIfbfguABYC1zabq2xlHwe2DBQtgX4VlWtAb7V5peKA8A/rao3AWcDv93+ny7VMT8PnFtVfwtYB2xIcja9W8Vc18b7LL1bySwVvwM82je/lMd60K9X1bq+y00X7P183AQCfbfgqKoXgIO3y1gyquq/0rvKq99G4OY2fTNw0YJ26giqqqer6v42/Ry9D44VLNExV8//abOvbK8CzqV3yxhYQuNNshJ4L/DZNh+W6FjnsGDv5+MpEGa7BceKQ9RdSv5GVT0NvQ9Q4PWL3J8jIskq4G3AvSzhMbdDKA8CzwB3Ak8AP6yqA63KUnpf/3vgnwEvtvlTWLpjPaiAP0pyX7tLAyzg+/l4eh5CZinzEqslIMkvAF8H/klV/UXvi+TS1H7Hsy7J64BvAG+ardrC9mr0krwPeKaq7kvyaweLZ6l6zI91wDuqal+S1wN3JvmfC7nx42kPocstOJaiP09yGkD7+8wi92ekkrySXhh8uar+cyte0mMGqKofAvfQO3fyunbLGFg67+t3ABcm2Uvv8O659PYYluJYf6aq9rW/z9AL/PUs4Pv5eAqE4/V2Gf23FbkC+OYi9mWk2jHlG4FHq+rf9S1akmNOMtb2DEjyauDd9M6b3E3vljGwRMZbVR+rqpVVtYrev9W7quoyluBYD0ry15L89YPTwPnAwyzg+/m4+mFakr9D71vGwdtlfGqRuzRSSb4K/Bq9OyT+OfAJ4A+BW4EzgCeBS6pq8MTzMSnJ3wb+G7CLvzrO/M/pnUdYcmNO8qv0Tiouo/dl7taq2prkDfS+RZ8MPAD8ZnsGyZLQDhl9pKret5TH2sb2jTZ7AvCVqvpUklNYoPfzcRUIkqRDO54OGUmSXoaBIEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAmA/w+YmIfVXf5f3gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pyplot.bar(range(len(xgb7_1.feature_importances_)),xgb7_1.feature_importances_)\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以比较明显的看出来，前面7个特征较为重要。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
